[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:13:25 dist-test-slave-pflr krb5kdc[24665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:13:25 dist-test-slave-pflr krb5kdc[24665](info): set up 2 sockets
Jun 20 20:13:25 dist-test-slave-pflr krb5kdc[24665](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 20:13:26.685035 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:13:26 dist-test-slave-pflr krb5kdc[24665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:13:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:13:26.805399 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:38577
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:38577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:26.910938 24678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:26.911190 24678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:26.911228 24678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:26.915016 24678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:26.915059 24678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:26.915077 24678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:26.915093 24678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:26.915108 24678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:26.920037 24678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:38577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:38577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:26.920967 24678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:26.921906 24678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:13:26.930156 24678 server_base.cc:1034] running on GCE node
W20260620 20:13:26.930305 24683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:26.930446 24684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:26.930406 24686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:26.931010 24678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:26.932144 24678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:26.933327 24678 hybrid_clock.cc:648] HybridClock initialized: now 1781986406933309 us; error 29 us; skew 500 ppm
Jun 20 20:13:26 dist-test-slave-pflr krb5kdc[24665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986406, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:26.936055 24678 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:13:26.936931 24678 webserver.cc:460] Webserver started at http://127.24.20.190:40589/ using document root <none> and password file <none>
I20260620 20:13:26.937434 24678 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:26.937489 24678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:26.937654 24678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:26.939545 24678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "a0e853401c75430e811453201ecaaacd"
format_stamp: "Formatted at 2026-06-20 20:13:26 on dist-test-slave-pflr"
server_key: "d3e40590f6eb0c41bdef70757be7ba70"
server_key_iv: "db855336769e61bb579d135bea06e4af"
server_key_version: "encryptionkey@0"
I20260620 20:13:26.940114 24678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "a0e853401c75430e811453201ecaaacd"
format_stamp: "Formatted at 2026-06-20 20:13:26 on dist-test-slave-pflr"
server_key: "d3e40590f6eb0c41bdef70757be7ba70"
server_key_iv: "db855336769e61bb579d135bea06e4af"
server_key_version: "encryptionkey@0"
I20260620 20:13:26.943608 24678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:13:26.945957 24693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:26.947156 24678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:26.947283 24678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "a0e853401c75430e811453201ecaaacd"
format_stamp: "Formatted at 2026-06-20 20:13:26 on dist-test-slave-pflr"
server_key: "d3e40590f6eb0c41bdef70757be7ba70"
server_key_iv: "db855336769e61bb579d135bea06e4af"
server_key_version: "encryptionkey@0"
I20260620 20:13:26.947377 24678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:26.958618 24678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:26.961800 24678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:26.961990 24678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:26.968722 24678 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:38577
I20260620 20:13:26.968757 24745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:38577 every 8 connection(s)
I20260620 20:13:26.969604 24678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:26.970762 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 24678
I20260620 20:13:26.970914 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:26.972191 24658 external_mini_cluster.cc:1428] Setting key f9ce2fbadcc1266b97c55a5f51cd905a
I20260620 20:13:26.972435 24746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:26.978471 24746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd: Bootstrap starting.
I20260620 20:13:26.980646 24746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:26.981357 24746 log.cc:824] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:26.983134 24746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd: No bootstrap required, opened a new log
I20260620 20:13:26.986485 24746 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e853401c75430e811453201ecaaacd" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 38577 } }
I20260620 20:13:26.986791 24746 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:26.986874 24746 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e853401c75430e811453201ecaaacd, State: Initialized, Role: FOLLOWER
I20260620 20:13:26.987299 24746 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e853401c75430e811453201ecaaacd" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 38577 } }
I20260620 20:13:26.987444 24746 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:26.987529 24746 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:26.987627 24746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:26.988452 24746 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e853401c75430e811453201ecaaacd" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 38577 } }
Jun 20 20:13:26 dist-test-slave-pflr krb5kdc[24665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:26.988814 24746 leader_election.cc:302] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0e853401c75430e811453201ecaaacd; no voters:
I20260620 20:13:26.989167 24746 leader_election.cc:288] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:26.989214 24751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:26.989403 24751 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [term 1 LEADER]: Becoming Leader. State: Replica: a0e853401c75430e811453201ecaaacd, State: Running, Role: LEADER
I20260620 20:13:26.989691 24751 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e853401c75430e811453201ecaaacd" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 38577 } }
I20260620 20:13:26.990373 24746 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:26.991335 24753 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0e853401c75430e811453201ecaaacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e853401c75430e811453201ecaaacd" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 38577 } } }
I20260620 20:13:26.991461 24753 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:26.991866 24760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:26.992909 24752 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0e853401c75430e811453201ecaaacd. Latest consensus state: current_term: 1 leader_uuid: "a0e853401c75430e811453201ecaaacd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0e853401c75430e811453201ecaaacd" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 38577 } } }
I20260620 20:13:26.993016 24752 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:26.995034 24760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:26.996078 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:26.975238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.24.20.190:38577)
0620 20:13:26.975622 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:26.975641 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:26.976680 (+ 1039us) server_negotiation.cc:407] Connection header received
0620 20:13:26.979712 (+ 3032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:26.979727 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:26.980025 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:26.980345 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:26.982512 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:26.983541 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:26.984751 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:26.985211 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:26.985383 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:26.989771 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:26.989780 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:26.989796 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:26.989828 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:26.992169 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:26.993136 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:26.993141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:26.993147 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:26.993300 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:26.993679 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:26.993682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:26.993684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:26.994163 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:26.994358 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:26.994896 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 20:13:26.995154 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":126,"threads_started":1}
I20260620 20:13:27.000996 24760 catalog_manager.cc:1347] Generated new cluster ID: feb144f4952f4e61b9e9264525ed2f50
I20260620 20:13:27.001070 24760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:27.022961 24760 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:27.023950 24760 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:27.036754 24760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0e853401c75430e811453201ecaaacd: Generated new TSK 0
I20260620 20:13:27.037602 24760 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:13:27.071853 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:38577
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:27.185489 24772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:27.185742 24772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:27.185786 24772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:27.189756 24772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:27.189810 24772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:27.189921 24772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:13:27.195204 24772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:38577
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:27.196199 24772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:27.197258 24772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:27.206414 24780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:27.206571 24772 server_base.cc:1034] running on GCE node
W20260620 20:13:27.206414 24777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:27.206455 24778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:27.207026 24772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:27.207662 24772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:27.208839 24772 hybrid_clock.cc:648] HybridClock initialized: now 1781986407208813 us; error 38 us; skew 500 ppm
Jun 20 20:13:27 dist-test-slave-pflr krb5kdc[24665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986407, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:27.211400 24772 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:13:27.212190 24772 webserver.cc:460] Webserver started at http://127.24.20.129:46575/ using document root <none> and password file <none>
I20260620 20:13:27.212785 24772 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:27.212869 24772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:27.213088 24772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:27.215047 24772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "68c53a98922249da9742d70385e04717"
format_stamp: "Formatted at 2026-06-20 20:13:27 on dist-test-slave-pflr"
server_key: "cba60a45e0d15ef6f9f5105ff29025ef"
server_key_iv: "4b79a4ae4ce585bafd5884c135618f14"
server_key_version: "encryptionkey@0"
I20260620 20:13:27.215601 24772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "68c53a98922249da9742d70385e04717"
format_stamp: "Formatted at 2026-06-20 20:13:27 on dist-test-slave-pflr"
server_key: "cba60a45e0d15ef6f9f5105ff29025ef"
server_key_iv: "4b79a4ae4ce585bafd5884c135618f14"
server_key_version: "encryptionkey@0"
I20260620 20:13:27.219148 24772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 20:13:27.221493 24787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:27.222610 24772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:27.222748 24772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "68c53a98922249da9742d70385e04717"
format_stamp: "Formatted at 2026-06-20 20:13:27 on dist-test-slave-pflr"
server_key: "cba60a45e0d15ef6f9f5105ff29025ef"
server_key_iv: "4b79a4ae4ce585bafd5884c135618f14"
server_key_version: "encryptionkey@0"
I20260620 20:13:27.222851 24772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:27.240952 24772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:27.243588 24772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:27.243793 24772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:27.244343 24772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:27.245193 24772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:27.245255 24772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:27.245321 24772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:27.245355 24772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:27.254155 24772 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:34481
I20260620 20:13:27.254199 24900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:34481 every 8 connection(s)
I20260620 20:13:27.254979 24772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:27.264206 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 24772
Jun 20 20:13:27 dist-test-slave-pflr krb5kdc[24665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986407, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:27.264321 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:13:27.264631 24658 external_mini_cluster.cc:1428] Setting key e18c206fcafb74dcd3df3a75d8ba0fc5
I20260620 20:13:27.268745 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:27.256704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:38073 (local address 127.24.20.190:38577)
0620 20:13:27.256835 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:27.256839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:27.257626 (+ 787us) server_negotiation.cc:407] Connection header received
0620 20:13:27.259213 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:27.259216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:27.259276 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:27.259385 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:27.260477 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.261116 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.262218 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.262496 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.262609 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:27.264979 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:27.264983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:27.264986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:27.265012 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:27.266667 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:27.267107 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:27.267113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:27.267118 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:27.267210 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:27.267504 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:27.267510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:27.267514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:27.267757 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:27.267861 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:27.268437 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 20:13:27.268550 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:13:27.269799 24903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:27.257056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:38577 (local address 127.24.20.129:38073)
0620 20:13:27.257476 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:27.257516 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:27.258982 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:27.259553 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:27.259560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:27.259894 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:27.260332 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:27.260344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.261237 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.261242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:27.262095 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:27.262103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.262641 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.262645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:27.262813 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:27.263307 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:27.263322 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:27.264816 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:27.266771 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:27.266775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:27.266784 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:27.267014 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:27.267296 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:27.267298 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:27.267300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:27.267419 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:27.267873 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:27.267881 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:27.268190 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 20:13:27.268396 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 20:13:27.268745 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260620 20:13:27.271049 24901 heartbeater.cc:347] Connected to a master server at 127.24.20.190:38577
I20260620 20:13:27.271298 24901 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:27.271864 24901 heartbeater.cc:511] Master 127.24.20.190:38577 requested a full tablet report, sending...
I20260620 20:13:27.273350 24710 ts_manager.cc:194] Registered new tserver with Master: 68c53a98922249da9742d70385e04717 (127.24.20.129:34481)
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:13:27.274502 24710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:38073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:13:27.290983 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:38577
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:27.397670 24906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:27.397899 24906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:27.397940 24906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:27.401749 24906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:27.401793 24906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:27.401846 24906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:13:27.406690 24906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:38577
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:27.407526 24906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:27.408386 24906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:27.417570 24912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:27.417691 24911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:27.417738 24906 server_base.cc:1034] running on GCE node
W20260620 20:13:27.417572 24914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:27.418185 24906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:27.418855 24906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:27.420035 24906 hybrid_clock.cc:648] HybridClock initialized: now 1781986407420022 us; error 27 us; skew 500 ppm
Jun 20 20:13:27 dist-test-slave-pflr krb5kdc[24665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986407, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:27.426955 24906 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:13:27.427810 24906 webserver.cc:460] Webserver started at http://127.24.20.130:33605/ using document root <none> and password file <none>
I20260620 20:13:27.428349 24906 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:27.428434 24906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:27.428711 24906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:27.430698 24906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "3a2d123437484a41bbf88afe75023ea9"
format_stamp: "Formatted at 2026-06-20 20:13:27 on dist-test-slave-pflr"
server_key: "fca03bc68b33db8e8976e53fcd86688d"
server_key_iv: "6260c573bb99531fb377b119384e217a"
server_key_version: "encryptionkey@0"
I20260620 20:13:27.431277 24906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "3a2d123437484a41bbf88afe75023ea9"
format_stamp: "Formatted at 2026-06-20 20:13:27 on dist-test-slave-pflr"
server_key: "fca03bc68b33db8e8976e53fcd86688d"
server_key_iv: "6260c573bb99531fb377b119384e217a"
server_key_version: "encryptionkey@0"
I20260620 20:13:27.434767 24906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:13:27.437037 24921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:27.438171 24906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:27.438298 24906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "3a2d123437484a41bbf88afe75023ea9"
format_stamp: "Formatted at 2026-06-20 20:13:27 on dist-test-slave-pflr"
server_key: "fca03bc68b33db8e8976e53fcd86688d"
server_key_iv: "6260c573bb99531fb377b119384e217a"
server_key_version: "encryptionkey@0"
I20260620 20:13:27.438402 24906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:27.451720 24906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:27.455049 24906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:27.455246 24906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:27.455790 24906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:27.456667 24906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:27.456717 24906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:27.456756 24906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:27.456777 24906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:27.465378 24906 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:32787
I20260620 20:13:27.465406 25034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:32787 every 8 connection(s)
I20260620 20:13:27.466279 24906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
Jun 20 20:13:27 dist-test-slave-pflr krb5kdc[24665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986407, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:27.476323 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 24906
I20260620 20:13:27.476437 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:13:27.476789 24658 external_mini_cluster.cc:1428] Setting key d68a11eca119f1a4a35ccf15e7ac42a7
I20260620 20:13:27.478811 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:27.467944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:37021 (local address 127.24.20.190:38577)
0620 20:13:27.468064 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:27.468067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:27.468864 (+ 797us) server_negotiation.cc:407] Connection header received
0620 20:13:27.470040 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:27.470043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:27.470088 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:27.470163 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:27.471185 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.471689 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.472802 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.473010 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.473087 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:27.475272 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:27.475275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:27.475277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:27.475299 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:27.476777 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:27.477235 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:27.477238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:27.477240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:27.477329 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:27.477682 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:27.477685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:27.477686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:27.477874 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:27.477975 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:27.478464 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 20:13:27.478584 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:13:27.479413 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:27.468300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:38577 (local address 127.24.20.130:37021)
0620 20:13:27.468723 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:27.468762 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:27.469857 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:27.470317 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:27.470324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:27.470641 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:27.471072 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:27.471083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.471786 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.471788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:27.472706 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:27.472715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.473124 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.473127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:27.473266 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:27.473769 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:27.473783 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:27.475128 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:27.476896 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:27.476902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:27.476921 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:27.477154 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:27.477428 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:27.477431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:27.477433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:27.477590 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:27.477990 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:27.477999 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:27.478242 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 20:13:27.478389 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 20:13:27.478608 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":111,"threads_started":1}
I20260620 20:13:27.480700 25035 heartbeater.cc:347] Connected to a master server at 127.24.20.190:38577
I20260620 20:13:27.480914 25035 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:27.481348 25035 heartbeater.cc:511] Master 127.24.20.190:38577 requested a full tablet report, sending...
I20260620 20:13:27.482386 24710 ts_manager.cc:194] Registered new tserver with Master: 3a2d123437484a41bbf88afe75023ea9 (127.24.20.130:32787)
I20260620 20:13:27.483248 24710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:37021
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:13:27.503316 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:38577
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:27.614459 25040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:27.614701 25040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:27.614766 25040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:27.618815 25040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:27.618886 25040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:27.618973 25040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:13:27.623977 25040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:38577
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:27.624979 25040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:27.625953 25040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:27.635289 25048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:27.635229 25046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:27.635329 25040 server_base.cc:1034] running on GCE node
W20260620 20:13:27.635229 25045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:27.635794 25040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:27.636369 25040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:27.637516 25040 hybrid_clock.cc:648] HybridClock initialized: now 1781986407637489 us; error 38 us; skew 500 ppm
Jun 20 20:13:27 dist-test-slave-pflr krb5kdc[24665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986407, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:27.640121 25040 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:13:27.640970 25040 webserver.cc:460] Webserver started at http://127.24.20.131:46339/ using document root <none> and password file <none>
I20260620 20:13:27.641495 25040 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:27.641552 25040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:27.641777 25040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:27.643800 25040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "a5fb602d79a5434696e2eb02ad893f6b"
format_stamp: "Formatted at 2026-06-20 20:13:27 on dist-test-slave-pflr"
server_key: "7c77abfec56b4c6c8fa965de5b45e7e3"
server_key_iv: "0325a3e67e653a0b32185da0077a37c6"
server_key_version: "encryptionkey@0"
I20260620 20:13:27.644354 25040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "a5fb602d79a5434696e2eb02ad893f6b"
format_stamp: "Formatted at 2026-06-20 20:13:27 on dist-test-slave-pflr"
server_key: "7c77abfec56b4c6c8fa965de5b45e7e3"
server_key_iv: "0325a3e67e653a0b32185da0077a37c6"
server_key_version: "encryptionkey@0"
I20260620 20:13:27.648037 25040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:13:27.650350 25055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:27.651607 25040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 20:13:27.651747 25040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "a5fb602d79a5434696e2eb02ad893f6b"
format_stamp: "Formatted at 2026-06-20 20:13:27 on dist-test-slave-pflr"
server_key: "7c77abfec56b4c6c8fa965de5b45e7e3"
server_key_iv: "0325a3e67e653a0b32185da0077a37c6"
server_key_version: "encryptionkey@0"
I20260620 20:13:27.651855 25040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:27.668757 25040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:27.671346 25040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:27.671535 25040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:27.672068 25040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:27.672885 25040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:27.672955 25040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:27.673018 25040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:27.673066 25040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:27.681530 25040 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:42451
I20260620 20:13:27.681567 25168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:42451 every 8 connection(s)
I20260620 20:13:27.682382 25040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:13:27.689209 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 25040
I20260620 20:13:27.689337 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:13:27.689600 24658 external_mini_cluster.cc:1428] Setting key 565d81d4ef416646a5834ff4716fcdc9
Jun 20 20:13:27 dist-test-slave-pflr krb5kdc[24665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986407, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:27.695735 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:27.684070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:43443 (local address 127.24.20.190:38577)
0620 20:13:27.684211 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:27.684214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:27.684982 (+ 768us) server_negotiation.cc:407] Connection header received
0620 20:13:27.686265 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:27.686269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:27.686332 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:27.686438 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:27.687541 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.688069 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.689198 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.689507 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.689617 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:27.692457 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:27.692460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:27.692462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:27.692484 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:27.693998 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:27.694435 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:27.694438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:27.694440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:27.694514 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:27.694852 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:27.694854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:27.694855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:27.695015 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:27.695089 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:27.695499 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 20:13:27.695624 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:13:27.696486 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:27.684407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:38577 (local address 127.24.20.131:43443)
0620 20:13:27.684837 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:27.684876 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:27.686047 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:27.686602 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:27.686608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:27.686936 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:27.687393 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:27.687406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.688198 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.688202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:27.689068 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:27.689077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.689650 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.689656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:27.689823 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:27.690406 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:27.690428 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:27.692320 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:27.694099 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:27.694103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:27.694113 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:27.694347 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:27.694647 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:27.694649 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:27.694650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:27.694772 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:27.695097 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:27.695101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:27.695302 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 20:13:27.695446 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 20:13:27.695677 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":139,"threads_started":1}
I20260620 20:13:27.697510 25169 heartbeater.cc:347] Connected to a master server at 127.24.20.190:38577
I20260620 20:13:27.697693 25169 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:27.698073 25169 heartbeater.cc:511] Master 127.24.20.190:38577 requested a full tablet report, sending...
I20260620 20:13:27.698951 24710 ts_manager.cc:194] Registered new tserver with Master: a5fb602d79a5434696e2eb02ad893f6b (127.24.20.131:42451)
I20260620 20:13:27.699445 24710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:43443
I20260620 20:13:27.703641 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:13:27 dist-test-slave-pflr krb5kdc[24665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986407, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:13:27 dist-test-slave-pflr krb5kdc[24665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986407, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.20.129@KRBTEST.COM
I20260620 20:13:27.727952 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:27.719064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.24.20.129:34481)
0620 20:13:27.719402 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:27.719411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:27.719434 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:13:27.719516 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:27.719521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:27.719701 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:27.719812 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:27.720408 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.721209 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.721964 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.722299 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.722376 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:27.724531 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:27.724540 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:27.724551 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:27.724573 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:27.726271 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:27.726692 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:27.726696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:27.726697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:27.726770 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:27.726982 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:27.726984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:27.726985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:27.727196 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:27.727392 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:27.727614 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 20:13:27.727765 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":156,"threads_started":1}
W20260620 20:13:27.728852 24815 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60552
Jun 20 20:13:27 dist-test-slave-pflr krb5kdc[24665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:13:27 dist-test-slave-pflr krb5kdc[24665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.129@KRBTEST.COM
I20260620 20:13:27.753376 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:27.745110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.24.20.129:34481)
0620 20:13:27.745284 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:27.745289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:27.745423 (+ 134us) server_negotiation.cc:407] Connection header received
0620 20:13:27.745578 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:27.745582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:27.745628 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:27.745715 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:27.746449 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.746971 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.747884 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:27.748115 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:27.748222 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:27.750476 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:27.750479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:27.750481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:27.750502 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:27.751780 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:27.752172 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:27.752176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:27.752178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:27.752255 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:27.752511 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:27.752516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:27.752518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:27.752729 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:27.752857 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:27.753083 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 20:13:27.753209 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:13:27.754272 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 24772
I20260620 20:13:27.760085 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 24906
I20260620 20:13:27.766923 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 25040
I20260620 20:13:27.772289 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 24678
2026-06-20T20:13:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2137 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:13:27 dist-test-slave-pflr krb5kdc[25183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:13:27 dist-test-slave-pflr krb5kdc[25183](info): set up 2 sockets
Jun 20 20:13:27 dist-test-slave-pflr krb5kdc[25183](info): commencing operation
krb5kdc: starting...
W20260620 20:13:28.812657 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:13:28 dist-test-slave-pflr krb5kdc[25183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:13:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:13:28.930902 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:37693
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:37693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:29.036875 25196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:29.037148 25196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:29.037235 25196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:29.041183 25196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:29.041257 25196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:29.041294 25196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:29.041350 25196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:29.041389 25196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:29.046624 25196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:37693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:37693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:29.047676 25196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:29.048676 25196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:29.057036 25204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:29.057036 25201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:29.057189 25202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:29.057297 25196 server_base.cc:1034] running on GCE node
I20260620 20:13:29.057864 25196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:29.058913 25196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:29.060079 25196 hybrid_clock.cc:648] HybridClock initialized: now 1781986409060037 us; error 50 us; skew 500 ppm
Jun 20 20:13:29 dist-test-slave-pflr krb5kdc[25183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:29.062888 25196 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:13:29.063710 25196 webserver.cc:460] Webserver started at http://127.24.20.190:44497/ using document root <none> and password file <none>
I20260620 20:13:29.064241 25196 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:29.064302 25196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:29.064507 25196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:29.066468 25196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "6b81489b9cef4c1aa6a7af3617bd814e"
format_stamp: "Formatted at 2026-06-20 20:13:29 on dist-test-slave-pflr"
server_key: "74430885592e1baf6d77f3ff0de74517"
server_key_iv: "68ef66aff71dde176ddbfe787284ecc1"
server_key_version: "encryptionkey@0"
I20260620 20:13:29.067020 25196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "6b81489b9cef4c1aa6a7af3617bd814e"
format_stamp: "Formatted at 2026-06-20 20:13:29 on dist-test-slave-pflr"
server_key: "74430885592e1baf6d77f3ff0de74517"
server_key_iv: "68ef66aff71dde176ddbfe787284ecc1"
server_key_version: "encryptionkey@0"
I20260620 20:13:29.070533 25196 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260620 20:13:29.072882 25211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:29.074069 25196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:13:29.074213 25196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "6b81489b9cef4c1aa6a7af3617bd814e"
format_stamp: "Formatted at 2026-06-20 20:13:29 on dist-test-slave-pflr"
server_key: "74430885592e1baf6d77f3ff0de74517"
server_key_iv: "68ef66aff71dde176ddbfe787284ecc1"
server_key_version: "encryptionkey@0"
I20260620 20:13:29.074321 25196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:29.089592 25196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:29.092813 25196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:29.093008 25196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:29.099361 25196 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:37693
I20260620 20:13:29.099412 25263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:37693 every 8 connection(s)
I20260620 20:13:29.100301 25196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:29.102972 25264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:29.106243 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 25196
I20260620 20:13:29.106353 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:29.106601 24658 external_mini_cluster.cc:1428] Setting key 5e6922af73043185475dd9d527cd6f3d
I20260620 20:13:29.108738 25264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e: Bootstrap starting.
I20260620 20:13:29.110785 25264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:29.111469 25264 log.cc:824] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:29.113229 25264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e: No bootstrap required, opened a new log
Jun 20 20:13:29 dist-test-slave-pflr krb5kdc[25183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:29.117189 25264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b81489b9cef4c1aa6a7af3617bd814e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37693 } }
I20260620 20:13:29.117390 25264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:29.117448 25264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b81489b9cef4c1aa6a7af3617bd814e, State: Initialized, Role: FOLLOWER
I20260620 20:13:29.117897 25264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b81489b9cef4c1aa6a7af3617bd814e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37693 } }
I20260620 20:13:29.118017 25264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:29.118098 25264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:29.118202 25264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:29.119251 25264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b81489b9cef4c1aa6a7af3617bd814e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37693 } }
I20260620 20:13:29.119619 25264 leader_election.cc:302] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b81489b9cef4c1aa6a7af3617bd814e; no voters:
I20260620 20:13:29.119947 25264 leader_election.cc:288] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:29.120056 25269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:29.120337 25269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [term 1 LEADER]: Becoming Leader. State: Replica: 6b81489b9cef4c1aa6a7af3617bd814e, State: Running, Role: LEADER
I20260620 20:13:29.120724 25269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b81489b9cef4c1aa6a7af3617bd814e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37693 } }
I20260620 20:13:29.121224 25264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:29.121543 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.108015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.24.20.190:37693)
0620 20:13:29.108487 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:29.108503 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:29.108538 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:13:29.109271 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:29.109288 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:29.109614 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:29.109944 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:29.110754 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.111856 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.112747 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.113201 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.113373 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.116058 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:29.116063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:29.116075 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:29.116100 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:29.117960 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:29.118609 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:29.118614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:29.118618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:29.118733 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:29.119045 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:29.119047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:29.119048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:29.119499 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:29.119692 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:29.120084 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 20:13:29.120646 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":136,"threads_started":1}
I20260620 20:13:29.122454 25271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b81489b9cef4c1aa6a7af3617bd814e. Latest consensus state: current_term: 1 leader_uuid: "6b81489b9cef4c1aa6a7af3617bd814e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b81489b9cef4c1aa6a7af3617bd814e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37693 } } }
I20260620 20:13:29.122496 25270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b81489b9cef4c1aa6a7af3617bd814e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b81489b9cef4c1aa6a7af3617bd814e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37693 } } }
I20260620 20:13:29.122567 25271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:29.122596 25270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e [sys.catalog]: This master's current role is: LEADER
W20260620 20:13:29.126049 25284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:13:29.126125 25284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:13:29.126186 25278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:29.126993 25278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:29.131844 25278 catalog_manager.cc:1347] Generated new cluster ID: f599273b4421453b9f964b4e1bb6f424
I20260620 20:13:29.131907 25278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:29.162786 25278 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:29.163764 25278 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:29.174464 25278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b81489b9cef4c1aa6a7af3617bd814e: Generated new TSK 0
I20260620 20:13:29.175179 25278 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:13:29.212148 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:37693
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:29.347381 25290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:29.347616 25290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:29.347658 25290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:29.351866 25290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:29.351920 25290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:29.352020 25290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:13:29.357180 25290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:37693
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:29.358196 25290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:29.359177 25290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:29.368340 25295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:29.368392 25290 server_base.cc:1034] running on GCE node
W20260620 20:13:29.368345 25298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:29.368359 25296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:29.369014 25290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:29.369594 25290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:29.370759 25290 hybrid_clock.cc:648] HybridClock initialized: now 1781986409370748 us; error 23 us; skew 500 ppm
Jun 20 20:13:29 dist-test-slave-pflr krb5kdc[25183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:29.373946 25290 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:13:29.374840 25290 webserver.cc:460] Webserver started at http://127.24.20.129:38415/ using document root <none> and password file <none>
I20260620 20:13:29.375418 25290 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:29.375485 25290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:29.375735 25290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:29.377938 25290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "5755f0c464f445d8b5edc03f0fcc8956"
format_stamp: "Formatted at 2026-06-20 20:13:29 on dist-test-slave-pflr"
server_key: "18058c07069f357385d9a6989af28dfc"
server_key_iv: "b1304e21a372b58cd31c7afd1d081267"
server_key_version: "encryptionkey@0"
I20260620 20:13:29.378551 25290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "5755f0c464f445d8b5edc03f0fcc8956"
format_stamp: "Formatted at 2026-06-20 20:13:29 on dist-test-slave-pflr"
server_key: "18058c07069f357385d9a6989af28dfc"
server_key_iv: "b1304e21a372b58cd31c7afd1d081267"
server_key_version: "encryptionkey@0"
I20260620 20:13:29.382391 25290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:13:29.384869 25305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:29.386067 25290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:29.386191 25290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "5755f0c464f445d8b5edc03f0fcc8956"
format_stamp: "Formatted at 2026-06-20 20:13:29 on dist-test-slave-pflr"
server_key: "18058c07069f357385d9a6989af28dfc"
server_key_iv: "b1304e21a372b58cd31c7afd1d081267"
server_key_version: "encryptionkey@0"
I20260620 20:13:29.386294 25290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:29.429721 25290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:29.433382 25290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:29.433610 25290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:29.434258 25290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:29.435159 25290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:29.435210 25290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:29.435284 25290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:29.435319 25290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:29.444656 25290 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:34321
I20260620 20:13:29.444716 25418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:34321 every 8 connection(s)
I20260620 20:13:29.445592 25290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:29.448753 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 25290
I20260620 20:13:29.448860 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:13:29.449141 24658 external_mini_cluster.cc:1428] Setting key 322fa62d2cb51f59aff38cb2b0d8a7d6
Jun 20 20:13:29 dist-test-slave-pflr krb5kdc[25183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:13:29.461222 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.447340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:38941 (local address 127.24.20.190:37693)
0620 20:13:29.447502 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:29.447507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:29.448345 (+ 838us) server_negotiation.cc:407] Connection header received
0620 20:13:29.449533 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:29.449537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:29.449599 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:29.449708 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:29.451082 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.452029 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.453808 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.454052 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.454142 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.457154 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:29.457158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:29.457161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:29.457191 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:29.458873 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:29.459428 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:29.459433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:29.459437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:29.459522 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:29.459904 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:29.459907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:29.459909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:29.460168 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:29.460249 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:29.460882 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 20:13:29.461019 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:13:29.462247 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.447752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:37693 (local address 127.24.20.129:38941)
0620 20:13:29.448196 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:29.448235 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:29.449273 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:29.449897 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:29.449908 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:29.450247 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:29.450936 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:29.450951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.452156 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.452159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:29.453681 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:29.453692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.454210 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.454217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:29.454442 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.455058 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:29.455084 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:29.456963 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:29.459008 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:29.459013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:29.459026 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:29.459306 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:29.459632 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:29.459635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:29.459636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:29.459783 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:29.460271 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:29.460276 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:29.460516 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 20:13:29.460817 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 20:13:29.461172 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
I20260620 20:13:29.463582 25419 heartbeater.cc:347] Connected to a master server at 127.24.20.190:37693
I20260620 20:13:29.463829 25419 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:29.464433 25419 heartbeater.cc:511] Master 127.24.20.190:37693 requested a full tablet report, sending...
I20260620 20:13:29.466023 25228 ts_manager.cc:194] Registered new tserver with Master: 5755f0c464f445d8b5edc03f0fcc8956 (127.24.20.129:34321)
I20260620 20:13:29.467263 25228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:38941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:13:29.478984 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:37693
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:29.586541 25424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:29.586807 25424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:29.586879 25424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:29.591081 25424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:29.591156 25424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:29.591240 25424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:13:29.596333 25424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:37693
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:29.597323 25424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:29.598271 25424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:13:29.607460 25424 server_base.cc:1034] running on GCE node
W20260620 20:13:29.607357 25429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:29.607442 25432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:29.607384 25430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:29.607988 25424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:29.608512 25424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:29.609694 25424 hybrid_clock.cc:648] HybridClock initialized: now 1781986409609679 us; error 25 us; skew 500 ppm
Jun 20 20:13:29 dist-test-slave-pflr krb5kdc[25183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:29.612156 25424 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:13:29.613045 25424 webserver.cc:460] Webserver started at http://127.24.20.130:43219/ using document root <none> and password file <none>
I20260620 20:13:29.613520 25424 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:29.613574 25424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:29.613732 25424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:29.615569 25424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "082b36d025664e89993f71a2a89f5838"
format_stamp: "Formatted at 2026-06-20 20:13:29 on dist-test-slave-pflr"
server_key: "a3b7bfc69689ced642a888668f786c68"
server_key_iv: "8a6d1a19039c2b3fbb1669a8a729d6dc"
server_key_version: "encryptionkey@0"
I20260620 20:13:29.616052 25424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "082b36d025664e89993f71a2a89f5838"
format_stamp: "Formatted at 2026-06-20 20:13:29 on dist-test-slave-pflr"
server_key: "a3b7bfc69689ced642a888668f786c68"
server_key_iv: "8a6d1a19039c2b3fbb1669a8a729d6dc"
server_key_version: "encryptionkey@0"
I20260620 20:13:29.619470 25424 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:13:29.621905 25439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:29.623359 25424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:13:29.623488 25424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "082b36d025664e89993f71a2a89f5838"
format_stamp: "Formatted at 2026-06-20 20:13:29 on dist-test-slave-pflr"
server_key: "a3b7bfc69689ced642a888668f786c68"
server_key_iv: "8a6d1a19039c2b3fbb1669a8a729d6dc"
server_key_version: "encryptionkey@0"
I20260620 20:13:29.623590 25424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:29.644766 25424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:29.647486 25424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:29.647687 25424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:29.648253 25424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:29.649094 25424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:29.649142 25424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:29.649210 25424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:29.649248 25424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:29.658401 25424 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:42439
I20260620 20:13:29.658421 25552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:42439 every 8 connection(s)
I20260620 20:13:29.659245 25424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:29.664763 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 25424
I20260620 20:13:29.664873 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:13:29.665146 24658 external_mini_cluster.cc:1428] Setting key 899d95ecbca3e4fc6882a24ca5524642
Jun 20 20:13:29 dist-test-slave-pflr krb5kdc[25183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:29.673005 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.661033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:53905 (local address 127.24.20.190:37693)
0620 20:13:29.661167 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:29.661170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:29.661999 (+ 829us) server_negotiation.cc:407] Connection header received
0620 20:13:29.662831 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:29.662833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:29.662878 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:29.662951 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:29.664012 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.664524 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.666071 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.666392 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.666500 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.669600 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:29.669603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:29.669606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:29.669627 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:29.671033 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:29.671517 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:29.671522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:29.671523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:29.671599 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:29.671934 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:29.671938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:29.671939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:29.672112 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:29.672217 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:29.672722 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 20:13:29.672837 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:13:29.673790 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.661421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:37693 (local address 127.24.20.130:53905)
0620 20:13:29.661847 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:29.661888 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:29.662636 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:29.663103 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:29.663110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:29.663436 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:29.663895 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:29.663907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.664682 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.664686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:29.665961 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:29.665969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.666554 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.666558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:29.666734 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.667368 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:29.667393 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:29.669408 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:29.671151 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:29.671157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:29.671170 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:29.671415 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:29.671698 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:29.671700 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:29.671703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:29.671833 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:29.672217 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:29.672224 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:29.672449 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 20:13:29.672643 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 20:13:29.672966 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
I20260620 20:13:29.674978 25553 heartbeater.cc:347] Connected to a master server at 127.24.20.190:37693
I20260620 20:13:29.675206 25553 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:29.675689 25553 heartbeater.cc:511] Master 127.24.20.190:37693 requested a full tablet report, sending...
I20260620 20:13:29.676815 25228 ts_manager.cc:194] Registered new tserver with Master: 082b36d025664e89993f71a2a89f5838 (127.24.20.130:42439)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
I20260620 20:13:29.677488 25228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:53905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:13:29.691339 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:37693
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:37611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:29.797288 25558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:29.797541 25558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:29.797585 25558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:29.801556 25558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:29.801611 25558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:29.801707 25558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:13:29.807173 25558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:37611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:37693
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:29.808157 25558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:29.809165 25558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:29.818260 25564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:29.818274 25566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:29.818285 25563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:29.818459 25558 server_base.cc:1034] running on GCE node
I20260620 20:13:29.818912 25558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:29.819540 25558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:29.820708 25558 hybrid_clock.cc:648] HybridClock initialized: now 1781986409820701 us; error 30 us; skew 500 ppm
Jun 20 20:13:29 dist-test-slave-pflr krb5kdc[25183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:29.823432 25558 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:13:29.824206 25558 webserver.cc:460] Webserver started at http://127.24.20.131:38087/ using document root <none> and password file <none>
I20260620 20:13:29.824801 25558 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:29.824884 25558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:29.825102 25558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:29.827172 25558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "04fd36311c4448a782e80335a7102677"
format_stamp: "Formatted at 2026-06-20 20:13:29 on dist-test-slave-pflr"
server_key: "f57fdb1ae22bc0e7933ad054b3d0d3de"
server_key_iv: "531d4d0c0b3ba57e646521dee2272016"
server_key_version: "encryptionkey@0"
I20260620 20:13:29.827759 25558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "04fd36311c4448a782e80335a7102677"
format_stamp: "Formatted at 2026-06-20 20:13:29 on dist-test-slave-pflr"
server_key: "f57fdb1ae22bc0e7933ad054b3d0d3de"
server_key_iv: "531d4d0c0b3ba57e646521dee2272016"
server_key_version: "encryptionkey@0"
I20260620 20:13:29.831238 25558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:13:29.833585 25573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:29.834740 25558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:29.834865 25558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "04fd36311c4448a782e80335a7102677"
format_stamp: "Formatted at 2026-06-20 20:13:29 on dist-test-slave-pflr"
server_key: "f57fdb1ae22bc0e7933ad054b3d0d3de"
server_key_iv: "531d4d0c0b3ba57e646521dee2272016"
server_key_version: "encryptionkey@0"
I20260620 20:13:29.834967 25558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:29.861578 25558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:29.864318 25558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:29.864512 25558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:29.865088 25558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:29.865871 25558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:29.865949 25558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:29.866017 25558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:29.866063 25558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:29.874910 25558 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:32915
I20260620 20:13:29.874934 25686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:32915 every 8 connection(s)
I20260620 20:13:29.875739 25558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:13:29.876904 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 25558
I20260620 20:13:29.877000 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:13:29.877303 24658 external_mini_cluster.cc:1428] Setting key df55f130c801eacdb910fa7e99faf9f4
Jun 20 20:13:29 dist-test-slave-pflr krb5kdc[25183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986409, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:29.888736 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.877608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:54219 (local address 127.24.20.190:37693)
0620 20:13:29.877763 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:29.877768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:29.878804 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 20:13:29.880169 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:29.880171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:29.880214 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:29.880267 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:29.881451 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.881958 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.883036 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.883237 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.883310 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.885678 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:29.885681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:29.885683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:29.885703 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:29.887095 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:29.887510 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:29.887512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:29.887514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:29.887581 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:29.887849 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:29.887852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:29.887853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:29.888006 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:29.888076 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:29.888477 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 20:13:29.888561 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:13:29.889472 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.878103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:37693 (local address 127.24.20.131:54219)
0620 20:13:29.878626 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:29.878711 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:29.879997 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:29.880538 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:29.880545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:29.880894 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:29.881337 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:29.881351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.882058 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.882061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:29.882942 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:29.882950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.883351 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.883353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:29.883533 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.884022 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:29.884036 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:29.885537 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:29.887187 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:29.887191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:29.887200 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:29.887430 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:29.887656 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:29.887658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:29.887660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:29.887775 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:29.888085 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:29.888090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:29.888294 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 20:13:29.888436 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 20:13:29.888718 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":130,"threads_started":1}
I20260620 20:13:29.890612 25687 heartbeater.cc:347] Connected to a master server at 127.24.20.190:37693
I20260620 20:13:29.890833 25687 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:29.891338 25687 heartbeater.cc:511] Master 127.24.20.190:37693 requested a full tablet report, sending...
I20260620 20:13:29.892241 25227 ts_manager.cc:194] Registered new tserver with Master: 04fd36311c4448a782e80335a7102677 (127.24.20.131:32915)
I20260620 20:13:29.892861 25227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:54219
I20260620 20:13:29.902472 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:13:29.912176 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.905779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.24.20.190:37693)
0620 20:13:29.905908 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:29.905911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:29.906011 (+ 100us) server_negotiation.cc:407] Connection header received
0620 20:13:29.906071 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:29.906073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:29.906111 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:29.906181 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:29.906826 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.907323 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.908059 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.908262 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.908332 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.909538 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:29.909541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:29.909543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:29.909563 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:29.910843 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:29.911215 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:29.911218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:29.911219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:29.911286 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:29.911501 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:29.911504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:29.911505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:29.911695 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:29.911792 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:29.911975 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 20:13:29.912058 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:13:29.916539 25227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:13:29.919443 25227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:29.933234 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.928110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.24.20.130:42439)
0620 20:13:29.928640 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:29.928652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:29.928674 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:13:29.928750 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:29.928756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:29.928994 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:29.929135 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:29.929588 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.930322 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.931819 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.932534 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.932682 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.932830 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:29.932994 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 20:13:29.933078 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":94,"threads_started":1}
I20260620 20:13:29.933223 25701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.927445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:42439 (local address 127.0.0.1:42558)
0620 20:13:29.928176 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:29.928188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:29.928262 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:29.929322 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:29.929326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:29.929343 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:29.929472 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:29.929478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.930452 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.930456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:29.931686 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:29.931697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.932693 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.932697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:29.932797 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.932832 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:13:29.932965 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 20:13:29.933077 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":24,"thread_start_us":93,"threads_started":1}
I20260620 20:13:29.933789 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.927343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:32915 (local address 127.0.0.1:53382)
0620 20:13:29.928009 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:29.928055 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:29.928187 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:29.929194 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:29.929198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:29.929217 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:29.929505 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:29.929512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.930947 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.930950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:29.932000 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:29.932008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.933288 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.933292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:29.933419 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.933505 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 20:13:29.933582 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 20:13:29.933663 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":180,"thread_start_us":65,"threads_started":1}
I20260620 20:13:29.933871 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.927459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.24.20.131:32915)
0620 20:13:29.928498 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:29.928513 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:29.928536 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:13:29.928648 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:29.928653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:29.928859 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:29.929007 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:29.929646 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.930812 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.932115 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.932964 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.933083 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.933226 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:29.933629 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 20:13:29.933738 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":176,"threads_started":1}
I20260620 20:13:29.933916 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.928122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.24.20.129:34321)
0620 20:13:29.928863 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:29.928896 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:29.928915 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:13:29.928988 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:29.928992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:29.929146 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:29.929279 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:29.929637 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.930776 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.932255 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.933032 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.933118 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.933226 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:29.933672 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 20:13:29.933759 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":89,"threads_started":1}
I20260620 20:13:29.934247 25699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:29.927695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:34321 (local address 127.0.0.1:56222)
0620 20:13:29.928490 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:29.928504 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:29.928592 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:29.929270 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:29.929273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:29.929285 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:29.929504 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:29.929512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.930918 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.930921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:29.932138 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:29.932148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:29.933288 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:29.933292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:29.933443 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:29.933516 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 20:13:29.933579 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 20:13:29.934098 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":650,"thread_start_us":88,"threads_started":1}
I20260620 20:13:29.936221 25487 tablet_service.cc:1467] Processing CreateTablet for tablet f8be02d6afd14090b48c9878a60a66c5 (DEFAULT_TABLE table=test-table [id=d7ccb4bf3a02402aad552be893044461]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:29.936491 25353 tablet_service.cc:1467] Processing CreateTablet for tablet f8be02d6afd14090b48c9878a60a66c5 (DEFAULT_TABLE table=test-table [id=d7ccb4bf3a02402aad552be893044461]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:29.937193 25487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8be02d6afd14090b48c9878a60a66c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:29.937492 25353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8be02d6afd14090b48c9878a60a66c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:29.938395 25621 tablet_service.cc:1467] Processing CreateTablet for tablet f8be02d6afd14090b48c9878a60a66c5 (DEFAULT_TABLE table=test-table [id=d7ccb4bf3a02402aad552be893044461]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:29.939298 25621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8be02d6afd14090b48c9878a60a66c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:29.943035 25704 tablet_bootstrap.cc:492] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838: Bootstrap starting.
I20260620 20:13:29.945071 25705 tablet_bootstrap.cc:492] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956: Bootstrap starting.
I20260620 20:13:29.945520 25704 tablet_bootstrap.cc:654] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:29.946020 25706 tablet_bootstrap.cc:492] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677: Bootstrap starting.
I20260620 20:13:29.946372 25704 log.cc:824] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:29.947127 25705 tablet_bootstrap.cc:654] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:29.947814 25706 tablet_bootstrap.cc:654] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:29.948289 25705 log.cc:824] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:29.948553 25704 tablet_bootstrap.cc:492] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838: No bootstrap required, opened a new log
I20260620 20:13:29.948778 25704 ts_tablet_manager.cc:1397] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:13:29.948870 25706 log.cc:824] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:29.950079 25705 tablet_bootstrap.cc:492] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956: No bootstrap required, opened a new log
I20260620 20:13:29.950259 25705 ts_tablet_manager.cc:1397] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 20:13:29.952539 25704 raft_consensus.cc:348] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04fd36311c4448a782e80335a7102677" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32915 } } peers { permanent_uuid: "082b36d025664e89993f71a2a89f5838" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42439 } } peers { permanent_uuid: "5755f0c464f445d8b5edc03f0fcc8956" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34321 } }
I20260620 20:13:29.952854 25704 raft_consensus.cc:374] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:29.952911 25704 raft_consensus.cc:729] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082b36d025664e89993f71a2a89f5838, State: Initialized, Role: FOLLOWER
I20260620 20:13:29.953348 25704 consensus_queue.cc:260] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04fd36311c4448a782e80335a7102677" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32915 } } peers { permanent_uuid: "082b36d025664e89993f71a2a89f5838" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42439 } } peers { permanent_uuid: "5755f0c464f445d8b5edc03f0fcc8956" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34321 } }
I20260620 20:13:29.953970 25553 heartbeater.cc:503] Master 127.24.20.190:37693 was elected leader, sending a full tablet report...
I20260620 20:13:29.953871 25705 raft_consensus.cc:348] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04fd36311c4448a782e80335a7102677" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32915 } } peers { permanent_uuid: "082b36d025664e89993f71a2a89f5838" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42439 } } peers { permanent_uuid: "5755f0c464f445d8b5edc03f0fcc8956" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34321 } }
I20260620 20:13:29.954130 25705 raft_consensus.cc:374] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:29.954202 25705 raft_consensus.cc:729] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5755f0c464f445d8b5edc03f0fcc8956, State: Initialized, Role: FOLLOWER
I20260620 20:13:29.954360 25704 ts_tablet_manager.cc:1428] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:13:29.954633 25705 consensus_queue.cc:260] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04fd36311c4448a782e80335a7102677" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32915 } } peers { permanent_uuid: "082b36d025664e89993f71a2a89f5838" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42439 } } peers { permanent_uuid: "5755f0c464f445d8b5edc03f0fcc8956" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34321 } }
I20260620 20:13:29.955572 25705 ts_tablet_manager.cc:1428] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 20:13:29.955891 25419 heartbeater.cc:503] Master 127.24.20.190:37693 was elected leader, sending a full tablet report...
I20260620 20:13:29.956079 25706 tablet_bootstrap.cc:492] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677: No bootstrap required, opened a new log
I20260620 20:13:29.956210 25706 ts_tablet_manager.cc:1397] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260620 20:13:29.959491 25706 raft_consensus.cc:348] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04fd36311c4448a782e80335a7102677" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32915 } } peers { permanent_uuid: "082b36d025664e89993f71a2a89f5838" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42439 } } peers { permanent_uuid: "5755f0c464f445d8b5edc03f0fcc8956" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34321 } }
I20260620 20:13:29.959689 25706 raft_consensus.cc:374] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:29.959764 25706 raft_consensus.cc:729] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fd36311c4448a782e80335a7102677, State: Initialized, Role: FOLLOWER
I20260620 20:13:29.960129 25706 consensus_queue.cc:260] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04fd36311c4448a782e80335a7102677" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32915 } } peers { permanent_uuid: "082b36d025664e89993f71a2a89f5838" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42439 } } peers { permanent_uuid: "5755f0c464f445d8b5edc03f0fcc8956" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34321 } }
I20260620 20:13:29.960718 25687 heartbeater.cc:503] Master 127.24.20.190:37693 was elected leader, sending a full tablet report...
I20260620 20:13:29.960955 25706 ts_tablet_manager.cc:1428] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:13:30.107714 25712 raft_consensus.cc:482] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:30.107949 25712 raft_consensus.cc:504] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04fd36311c4448a782e80335a7102677" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32915 } } peers { permanent_uuid: "082b36d025664e89993f71a2a89f5838" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42439 } } peers { permanent_uuid: "5755f0c464f445d8b5edc03f0fcc8956" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34321 } }
I20260620 20:13:30.109174 25712 leader_election.cc:288] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 082b36d025664e89993f71a2a89f5838 (127.24.20.130:42439), 5755f0c464f445d8b5edc03f0fcc8956 (127.24.20.129:34321)
I20260620 20:13:30.110040 25711 raft_consensus.cc:482] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:30.110306 25711 raft_consensus.cc:504] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04fd36311c4448a782e80335a7102677" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32915 } } peers { permanent_uuid: "082b36d025664e89993f71a2a89f5838" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42439 } } peers { permanent_uuid: "5755f0c464f445d8b5edc03f0fcc8956" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34321 } }
I20260620 20:13:30.111855 25711 leader_election.cc:288] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04fd36311c4448a782e80335a7102677 (127.24.20.131:32915), 082b36d025664e89993f71a2a89f5838 (127.24.20.130:42439)
I20260620 20:13:30.113723 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:30.109368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:42439 (local address 127.24.20.131:51593)
0620 20:13:30.109551 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:30.109567 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:30.109675 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:30.110062 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:30.110067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:30.110086 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:30.110217 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:30.110223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.111216 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.111219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:30.112371 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:30.112382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.113315 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.113319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:30.113426 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:30.113454 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 20:13:30.113518 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 20:13:30.113579 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"spinlock_wait_cycles":3968}
I20260620 20:13:30.114359 25702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:30.109529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:51593 (local address 127.24.20.130:42439)
0620 20:13:30.109641 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:30.109646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:30.109661 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:13:30.109815 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:30.109818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:30.109868 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:30.109938 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:30.110355 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.111083 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.112510 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.113146 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.113258 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:30.114142 (+ 884us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:30.114175 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 20:13:30.114239 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":818,"server-negotiator.queue_time_us":47}
I20260620 20:13:30.114372 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:30.109838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:41581 (local address 127.24.20.129:34321)
0620 20:13:30.109955 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:30.109959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:30.110046 (+ 87us) server_negotiation.cc:407] Connection header received
0620 20:13:30.110218 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:30.110221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:30.110269 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:30.110314 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:30.110676 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.111412 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.113124 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.113700 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.113798 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:30.113848 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:30.114177 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 20:13:30.114256 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:13:30.114706 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:30.109596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:34321 (local address 127.24.20.131:41581)
0620 20:13:30.109958 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:30.109972 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:30.110113 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:30.110438 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:30.110441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:30.110454 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:30.110560 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:30.110566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.111562 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.111564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:30.113010 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:30.113018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.113948 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.113951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:30.114043 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:30.114064 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:13:30.114502 (+ 438us) client_negotiation.cc:240] Negotiation successful
0620 20:13:30.114569 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":106,"threads_started":1}
I20260620 20:13:30.114895 25507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8be02d6afd14090b48c9878a60a66c5" candidate_uuid: "04fd36311c4448a782e80335a7102677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "082b36d025664e89993f71a2a89f5838" is_pre_election: true
I20260620 20:13:30.115157 25507 raft_consensus.cc:2454] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04fd36311c4448a782e80335a7102677 in term 0.
I20260620 20:13:30.115371 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8be02d6afd14090b48c9878a60a66c5" candidate_uuid: "04fd36311c4448a782e80335a7102677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5755f0c464f445d8b5edc03f0fcc8956" is_pre_election: true
I20260620 20:13:30.115550 25373 raft_consensus.cc:2454] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04fd36311c4448a782e80335a7102677 in term 0.
I20260620 20:13:30.115717 25574 leader_election.cc:302] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04fd36311c4448a782e80335a7102677, 082b36d025664e89993f71a2a89f5838; no voters:
I20260620 20:13:30.116110 25712 raft_consensus.cc:2781] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:30.116181 25712 raft_consensus.cc:482] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:30.116221 25712 raft_consensus.cc:3037] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:30.116343 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:30.111504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:32915 (local address 127.24.20.129:35059)
0620 20:13:30.112291 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:30.112304 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:30.112423 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:30.112740 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:30.112743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:30.112759 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:30.112857 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:30.112862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.113828 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.113831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:30.115012 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:30.115024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.115890 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.115893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:30.116016 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:30.116034 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 20:13:30.116127 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 20:13:30.116190 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":72,"threads_started":1}
I20260620 20:13:30.116357 25700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:30.111801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:35059 (local address 127.24.20.131:32915)
0620 20:13:30.111903 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:30.111909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:30.112365 (+ 456us) server_negotiation.cc:407] Connection header received
0620 20:13:30.112521 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:30.112524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:30.112559 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:30.112622 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:30.112977 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.113685 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.115140 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.115666 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.115752 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:30.115805 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:30.116143 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 20:13:30.116201 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:13:30.117004 25715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:30.112059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:42439 (local address 127.24.20.129:33251)
0620 20:13:30.112544 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:30.112557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:30.112707 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:30.113010 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:30.113012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:30.113026 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:30.113130 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:30.113135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.114163 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.114166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:30.115622 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:30.115632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.116583 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.116608 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:30.116695 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:30.116714 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 20:13:30.116820 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:13:30.116876 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":83,"threads_started":1}
I20260620 20:13:30.116951 25641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8be02d6afd14090b48c9878a60a66c5" candidate_uuid: "5755f0c464f445d8b5edc03f0fcc8956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04fd36311c4448a782e80335a7102677" is_pre_election: true
I20260620 20:13:30.117079 25716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:30.112105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:33251 (local address 127.24.20.130:42439)
0620 20:13:30.112371 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:30.112379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:30.112646 (+ 267us) server_negotiation.cc:407] Connection header received
0620 20:13:30.112808 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:30.112811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:30.112853 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:30.112902 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:30.113255 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.114013 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.115752 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.116367 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.116484 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:30.116523 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:30.116872 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 20:13:30.116951 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":210,"thread_start_us":98,"threads_started":1}
I20260620 20:13:30.117646 25507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8be02d6afd14090b48c9878a60a66c5" candidate_uuid: "5755f0c464f445d8b5edc03f0fcc8956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "082b36d025664e89993f71a2a89f5838" is_pre_election: true
I20260620 20:13:30.117655 25712 raft_consensus.cc:504] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04fd36311c4448a782e80335a7102677" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32915 } } peers { permanent_uuid: "082b36d025664e89993f71a2a89f5838" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42439 } } peers { permanent_uuid: "5755f0c464f445d8b5edc03f0fcc8956" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34321 } }
I20260620 20:13:30.117744 25507 raft_consensus.cc:2454] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5755f0c464f445d8b5edc03f0fcc8956 in term 0.
I20260620 20:13:30.117929 25641 raft_consensus.cc:2379] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5755f0c464f445d8b5edc03f0fcc8956 in current term 1: Already voted for candidate 04fd36311c4448a782e80335a7102677 in this term.
I20260620 20:13:30.118029 25712 leader_election.cc:288] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [CANDIDATE]: Term 1 election: Requested vote from peers 082b36d025664e89993f71a2a89f5838 (127.24.20.130:42439), 5755f0c464f445d8b5edc03f0fcc8956 (127.24.20.129:34321)
I20260620 20:13:30.118137 25306 leader_election.cc:302] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082b36d025664e89993f71a2a89f5838, 5755f0c464f445d8b5edc03f0fcc8956; no voters:
I20260620 20:13:30.118372 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8be02d6afd14090b48c9878a60a66c5" candidate_uuid: "04fd36311c4448a782e80335a7102677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5755f0c464f445d8b5edc03f0fcc8956"
I20260620 20:13:30.118470 25373 raft_consensus.cc:3037] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:30.118753 25507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8be02d6afd14090b48c9878a60a66c5" candidate_uuid: "04fd36311c4448a782e80335a7102677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "082b36d025664e89993f71a2a89f5838"
I20260620 20:13:30.118849 25507 raft_consensus.cc:3037] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:30.119560 25373 raft_consensus.cc:2454] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04fd36311c4448a782e80335a7102677 in term 1.
I20260620 20:13:30.119669 25711 raft_consensus.cc:2741] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 20:13:30.119905 25577 leader_election.cc:302] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04fd36311c4448a782e80335a7102677, 5755f0c464f445d8b5edc03f0fcc8956; no voters:
I20260620 20:13:30.120016 25507 raft_consensus.cc:2454] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04fd36311c4448a782e80335a7102677 in term 1.
I20260620 20:13:30.120201 25712 raft_consensus.cc:2781] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:30.120416 25712 raft_consensus.cc:686] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [term 1 LEADER]: Becoming Leader. State: Replica: 04fd36311c4448a782e80335a7102677, State: Running, Role: LEADER
I20260620 20:13:30.120754 25712 consensus_queue.cc:237] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04fd36311c4448a782e80335a7102677" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32915 } } peers { permanent_uuid: "082b36d025664e89993f71a2a89f5838" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42439 } } peers { permanent_uuid: "5755f0c464f445d8b5edc03f0fcc8956" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34321 } }
I20260620 20:13:30.123811 25227 catalog_manager.cc:5495] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04fd36311c4448a782e80335a7102677 (127.24.20.131). New cstate: current_term: 1 leader_uuid: "04fd36311c4448a782e80335a7102677" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04fd36311c4448a782e80335a7102677" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "082b36d025664e89993f71a2a89f5838" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5755f0c464f445d8b5edc03f0fcc8956" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34321 } health_report { overall_health: UNKNOWN } } }
W20260620 20:13:30.126538 25688 tablet.cc:2369] T f8be02d6afd14090b48c9878a60a66c5 P 04fd36311c4448a782e80335a7102677: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:13:30.160408 25554 tablet.cc:2369] T f8be02d6afd14090b48c9878a60a66c5 P 082b36d025664e89993f71a2a89f5838: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:13:30 dist-test-slave-pflr krb5kdc[25183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.129@KRBTEST.COM
I20260620 20:13:30.174166 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:30.166074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.24.20.129:34321)
0620 20:13:30.166249 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:30.166253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:30.166340 (+ 87us) server_negotiation.cc:407] Connection header received
0620 20:13:30.166443 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:30.166446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:30.166485 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:30.166550 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:30.167171 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.167669 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.168432 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.168706 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.168785 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:30.171121 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:30.171131 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:30.171148 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:30.171170 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:30.172709 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:30.173066 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:30.173071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:30.173072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:30.173149 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:30.173362 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:30.173364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:30.173365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:30.173579 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:30.173741 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:30.173862 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 20:13:30.173950 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 20:13:30 dist-test-slave-pflr krb5kdc[25183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986410, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:13:30 dist-test-slave-pflr krb5kdc[25183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986410, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.20.129@KRBTEST.COM
W20260620 20:13:30.196880 25420 tablet.cc:2369] T f8be02d6afd14090b48c9878a60a66c5 P 5755f0c464f445d8b5edc03f0fcc8956: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:13:30.199549 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:30.191710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.24.20.129:34321)
0620 20:13:30.191868 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:30.191874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:30.191999 (+ 125us) server_negotiation.cc:407] Connection header received
0620 20:13:30.192178 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:30.192182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:30.192223 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:30.192310 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:30.193058 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.193594 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.194431 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.194665 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.194774 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:30.196988 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:30.196992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:30.196995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:30.197018 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:30.198274 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:30.198618 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:30.198620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:30.198621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:30.198685 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:30.198893 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:30.198895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:30.198896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:30.199033 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:30.199124 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:30.199352 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 20:13:30.199440 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 20:13:30 dist-test-slave-pflr krb5kdc[25183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:13:30 dist-test-slave-pflr krb5kdc[25183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.129@KRBTEST.COM
I20260620 20:13:30.223304 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:30.215800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.24.20.129:34321)
0620 20:13:30.215954 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:30.215957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:30.216016 (+ 59us) server_negotiation.cc:407] Connection header received
0620 20:13:30.216126 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:30.216128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:30.216167 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:30.216235 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:30.216893 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.217372 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.218158 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:30.218350 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:30.218422 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:30.220568 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:30.220571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:30.220573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:30.220674 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:30.221931 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:30.222269 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:30.222271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:30.222273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:30.222339 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:30.222558 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:30.222561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:30.222562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:30.222710 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:30.222856 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:30.223005 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 20:13:30.223094 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 20:13:30.224529 25333 tablet_service.cc:2908] Rejecting scan request for tablet f8be02d6afd14090b48c9878a60a66c5: Uninitialized: clean time has not yet been initialized
I20260620 20:13:30.226383 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 25290
I20260620 20:13:30.231942 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 25424
I20260620 20:13:30.237103 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 25558
I20260620 20:13:30.242663 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 25196
2026-06-20T20:13:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2470 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:13:30 dist-test-slave-pflr krb5kdc[25731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:13:30 dist-test-slave-pflr krb5kdc[25731](info): set up 2 sockets
Jun 20 20:13:30 dist-test-slave-pflr krb5kdc[25731](info): commencing operation
krb5kdc: starting...
W20260620 20:13:31.272516 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:13:31 dist-test-slave-pflr krb5kdc[25731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:13:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:13:31.389840 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:39683
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:39683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:31.496141 25744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:31.496447 25744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:31.496538 25744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:31.500526 25744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:31.500648 25744 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:13:31.500694 25744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:31.500748 25744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:31.500784 25744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:31.500823 25744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:31.506224 25744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:39683
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:39683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:31.507328 25744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:31.508375 25744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:31.517079 25752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:31.517064 25749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:31.517062 25750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:31.517612 25744 server_base.cc:1034] running on GCE node
I20260620 20:13:31.518167 25744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:31.519251 25744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:31.520442 25744 hybrid_clock.cc:648] HybridClock initialized: now 1781986411520423 us; error 33 us; skew 500 ppm
Jun 20 20:13:31 dist-test-slave-pflr krb5kdc[25731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986411, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:31.523193 25744 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:13:31.524086 25744 webserver.cc:460] Webserver started at http://127.24.20.190:34955/ using document root <none> and password file <none>
I20260620 20:13:31.524667 25744 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:31.524730 25744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:31.524900 25744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:31.526818 25744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "6f115ded05ac4194be4a93d4051765f8"
format_stamp: "Formatted at 2026-06-20 20:13:31 on dist-test-slave-pflr"
server_key: "43a6e1f293e0aa07afb94d8f81b1f377"
server_key_iv: "784d98d3f0fa635b616d8a79b39247aa"
server_key_version: "encryptionkey@0"
I20260620 20:13:31.527310 25744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "6f115ded05ac4194be4a93d4051765f8"
format_stamp: "Formatted at 2026-06-20 20:13:31 on dist-test-slave-pflr"
server_key: "43a6e1f293e0aa07afb94d8f81b1f377"
server_key_iv: "784d98d3f0fa635b616d8a79b39247aa"
server_key_version: "encryptionkey@0"
I20260620 20:13:31.530843 25744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:13:31.533171 25759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:31.534345 25744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:31.534471 25744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "6f115ded05ac4194be4a93d4051765f8"
format_stamp: "Formatted at 2026-06-20 20:13:31 on dist-test-slave-pflr"
server_key: "43a6e1f293e0aa07afb94d8f81b1f377"
server_key_iv: "784d98d3f0fa635b616d8a79b39247aa"
server_key_version: "encryptionkey@0"
I20260620 20:13:31.534572 25744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:31.547390 25744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:31.550550 25744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:31.550742 25744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:31.558753 25744 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:39683
I20260620 20:13:31.558781 25821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:39683 every 8 connection(s)
I20260620 20:13:31.559679 25744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:31.562410 25822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:31.565176 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 25744
I20260620 20:13:31.565302 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:31.565629 24658 external_mini_cluster.cc:1428] Setting key 698ccbd8b9ca802d859367a5ab9bd95d
I20260620 20:13:31.569042 25822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8: Bootstrap starting.
I20260620 20:13:31.571730 25822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:31.572468 25822 log.cc:824] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:31.574456 25822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8: No bootstrap required, opened a new log
Jun 20 20:13:31 dist-test-slave-pflr krb5kdc[25731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:31.578715 25822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f115ded05ac4194be4a93d4051765f8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39683 } }
I20260620 20:13:31.578944 25822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:31.579015 25822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f115ded05ac4194be4a93d4051765f8, State: Initialized, Role: FOLLOWER
I20260620 20:13:31.579483 25822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f115ded05ac4194be4a93d4051765f8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39683 } }
I20260620 20:13:31.579630 25822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:31.579699 25822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:31.579802 25822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:31.580694 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:31.567290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.24.20.190:39683)
0620 20:13:31.567823 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:31.567843 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:31.567885 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:13:31.568698 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:31.568722 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:31.569127 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:31.569499 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:31.570151 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:31.571622 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:31.572506 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:31.573107 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:31.573296 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:31.575975 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:31.575980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:31.575992 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:31.576016 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:31.577721 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:31.578198 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:31.578202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:31.578206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:31.578315 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:31.578612 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:31.578614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:31.578615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:31.579080 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:31.579301 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:31.579574 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 20:13:31.579884 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":130,"threads_started":1}
I20260620 20:13:31.580693 25822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f115ded05ac4194be4a93d4051765f8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39683 } }
I20260620 20:13:31.581101 25822 leader_election.cc:302] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f115ded05ac4194be4a93d4051765f8; no voters:
I20260620 20:13:31.581422 25822 leader_election.cc:288] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:31.581497 25827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:31.581708 25827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [term 1 LEADER]: Becoming Leader. State: Replica: 6f115ded05ac4194be4a93d4051765f8, State: Running, Role: LEADER
I20260620 20:13:31.582039 25827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f115ded05ac4194be4a93d4051765f8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39683 } }
I20260620 20:13:31.582679 25822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:31.583202 25829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f115ded05ac4194be4a93d4051765f8. Latest consensus state: current_term: 1 leader_uuid: "6f115ded05ac4194be4a93d4051765f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f115ded05ac4194be4a93d4051765f8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39683 } } }
I20260620 20:13:31.583189 25828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f115ded05ac4194be4a93d4051765f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f115ded05ac4194be4a93d4051765f8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39683 } } }
I20260620 20:13:31.583339 25828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:31.583526 25829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8 [sys.catalog]: This master's current role is: LEADER
W20260620 20:13:31.587216 25842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:13:31.587307 25842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:13:31.587378 25832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:31.588049 25832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:31.593592 25832 catalog_manager.cc:1347] Generated new cluster ID: 8d17e1164f95414d87ff07b771262700
I20260620 20:13:31.593660 25832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:31.618999 25832 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:31.620054 25832 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:31.624918 25832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8: Generated new TSK 0
I20260620 20:13:31.625703 25832 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:13:31.656773 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:39683
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:31.770218 25848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:31.770467 25848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:31.770541 25848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:31.774600 25848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:31.774655 25848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:31.774729 25848 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:13:31.774814 25848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:13:31.779891 25848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.20.190:39683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:31.780896 25848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:31.781886 25848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:31.791603 25854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:31.791599 25853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:31.791672 25856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:31.791882 25848 server_base.cc:1034] running on GCE node
I20260620 20:13:31.792342 25848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:31.793017 25848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:31.794219 25848 hybrid_clock.cc:648] HybridClock initialized: now 1781986411794207 us; error 32 us; skew 500 ppm
Jun 20 20:13:31 dist-test-slave-pflr krb5kdc[25731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986411, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:31.797165 25848 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:13:31.798048 25848 webserver.cc:460] Webserver started at http://127.24.20.129:46643/ using document root <none> and password file <none>
I20260620 20:13:31.798614 25848 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:31.798704 25848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:31.798933 25848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:31.801075 25848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "e01402125ffe43aea3679f5dc72ff511"
format_stamp: "Formatted at 2026-06-20 20:13:31 on dist-test-slave-pflr"
server_key: "5b277001d843ccdfbef40a7eb74014af"
server_key_iv: "01c5aa04bb7e16cd46ffb87bc8f7c08e"
server_key_version: "encryptionkey@0"
I20260620 20:13:31.801641 25848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "e01402125ffe43aea3679f5dc72ff511"
format_stamp: "Formatted at 2026-06-20 20:13:31 on dist-test-slave-pflr"
server_key: "5b277001d843ccdfbef40a7eb74014af"
server_key_iv: "01c5aa04bb7e16cd46ffb87bc8f7c08e"
server_key_version: "encryptionkey@0"
I20260620 20:13:31.805464 25848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 20:13:31.807947 25863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:31.809319 25848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:13:31.809523 25848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "e01402125ffe43aea3679f5dc72ff511"
format_stamp: "Formatted at 2026-06-20 20:13:31 on dist-test-slave-pflr"
server_key: "5b277001d843ccdfbef40a7eb74014af"
server_key_iv: "01c5aa04bb7e16cd46ffb87bc8f7c08e"
server_key_version: "encryptionkey@0"
I20260620 20:13:31.809661 25848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:31.836323 25848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:31.839583 25848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:31.839807 25848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:31.841344 25848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:31.841416 25848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:31.841522 25848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:31.841560 25848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:13:31 dist-test-slave-pflr krb5kdc[25731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986411, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:31.852445 25848 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:38045
I20260620 20:13:31.852720 25978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:38045 every 8 connection(s)
I20260620 20:13:31.853574 25848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:31.855999 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:31.842003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:41771 (local address 127.24.20.190:39683)
0620 20:13:31.842156 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:31.842161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:31.843486 (+ 1325us) server_negotiation.cc:407] Connection header received
0620 20:13:31.844745 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:31.844749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:31.844801 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:31.844852 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:31.846545 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:31.847117 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:31.848447 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:31.848897 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:31.849017 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:31.852028 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:31.852032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:31.852034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:31.852059 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:31.853966 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:31.854640 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:31.854644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:31.854646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:31.854739 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:31.855035 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:31.855037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:31.855038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:31.855241 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:31.855322 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:31.855776 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 20:13:31.855879 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:13:31.857007 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:31.842876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:39683 (local address 127.24.20.129:41771)
0620 20:13:31.843329 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:31.843376 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:31.844462 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:31.845238 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:31.845246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:31.845739 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:31.846272 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:31.846285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:31.847251 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:31.847255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:31.848301 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:31.848311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:31.849271 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:31.849276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:31.849538 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:31.850169 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:31.850186 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:31.851841 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:31.854104 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:31.854109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:31.854123 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:31.854529 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:31.854828 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:31.854830 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:31.854831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:31.854954 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:31.855328 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:31.855333 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:31.855575 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 20:13:31.855729 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 20:13:31.855962 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":124,"threads_started":1}
I20260620 20:13:31.858315 25979 heartbeater.cc:347] Connected to a master server at 127.24.20.190:39683
I20260620 20:13:31.858451 25979 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:31.858999 25979 heartbeater.cc:511] Master 127.24.20.190:39683 requested a full tablet report, sending...
I20260620 20:13:31.860684 25775 ts_manager.cc:194] Registered new tserver with Master: e01402125ffe43aea3679f5dc72ff511 (127.24.20.129:38045)
I20260620 20:13:31.861789 25775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:41771
I20260620 20:13:31.863384 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 25848
I20260620 20:13:31.863497 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:13:31.863786 24658 external_mini_cluster.cc:1428] Setting key 710d5a2bf269e6f594de20549d6a3e85
I20260620 20:13:31.869160 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:31.860586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:39683 (local address 127.24.20.129:45801)
0620 20:13:31.860891 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:31.860911 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:31.861020 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:31.861244 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:31.861246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:31.861439 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:31.861581 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:31.861587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:31.862371 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:31.862375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:31.863061 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:31.863070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:31.864520 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:31.864524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:31.864660 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:31.865222 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:31.865233 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:31.865642 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:31.867735 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:31.867738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:31.867739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:31.867930 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:31.868222 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:31.868224 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:31.868226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:31.868297 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:31.868744 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:31.868746 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:31.868819 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 20:13:31.868903 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 20:13:31.869001 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":7,"thread_start_us":93,"threads_started":1}
I20260620 20:13:31.869371 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:31.860697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:45801 (local address 127.24.20.190:39683)
0620 20:13:31.860817 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:31.860821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:31.860985 (+ 164us) server_negotiation.cc:407] Connection header received
0620 20:13:31.861094 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:31.861097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:31.861138 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:31.861229 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:31.861692 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:31.862202 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:31.864183 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:31.864386 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:31.864508 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:31.865875 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:31.865878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:31.865881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:31.865903 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:31.867620 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:31.868034 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:31.868041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:31.868046 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:31.868130 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:31.868385 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:31.868391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:31.868396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:31.868650 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:31.868752 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:31.869007 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 20:13:31.869139 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:13:31.892181 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:39683
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:32.000699 25988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:32.000936 25988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:32.000978 25988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:32.004952 25988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:32.005009 25988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:32.005048 25988 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:13:32.005123 25988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:13:32.010264 25988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.20.190:39683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:32.011260 25988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:32.012305 25988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:32.021768 25994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:32.021778 25993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:32.021888 25988 server_base.cc:1034] running on GCE node
W20260620 20:13:32.021773 25996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:32.022437 25988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:32.023106 25988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:32.024291 25988 hybrid_clock.cc:648] HybridClock initialized: now 1781986412024276 us; error 27 us; skew 500 ppm
Jun 20 20:13:32 dist-test-slave-pflr krb5kdc[25731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986412, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:32.027204 25988 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:13:32.028088 25988 webserver.cc:460] Webserver started at http://127.24.20.130:35341/ using document root <none> and password file <none>
I20260620 20:13:32.028680 25988 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:32.028769 25988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:32.029006 25988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:32.030974 25988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "97ef923e61924e55ae2ded08b5f5ffde"
format_stamp: "Formatted at 2026-06-20 20:13:32 on dist-test-slave-pflr"
server_key: "be10e173f27f59391abd1ef4e1f0d202"
server_key_iv: "225be8c1599bd1dac68fa21097871440"
server_key_version: "encryptionkey@0"
I20260620 20:13:32.031549 25988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "97ef923e61924e55ae2ded08b5f5ffde"
format_stamp: "Formatted at 2026-06-20 20:13:32 on dist-test-slave-pflr"
server_key: "be10e173f27f59391abd1ef4e1f0d202"
server_key_iv: "225be8c1599bd1dac68fa21097871440"
server_key_version: "encryptionkey@0"
I20260620 20:13:32.035143 25988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 20:13:32.037472 26003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:32.038789 25988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:32.038910 25988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "97ef923e61924e55ae2ded08b5f5ffde"
format_stamp: "Formatted at 2026-06-20 20:13:32 on dist-test-slave-pflr"
server_key: "be10e173f27f59391abd1ef4e1f0d202"
server_key_iv: "225be8c1599bd1dac68fa21097871440"
server_key_version: "encryptionkey@0"
I20260620 20:13:32.039014 25988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:32.059358 25988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:32.062717 25988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:32.062924 25988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:32.064394 25988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:32.064479 25988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:32.064548 25988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:32.064620 25988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:13:32 dist-test-slave-pflr krb5kdc[25731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986412, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:32.075562 25988 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:40919
I20260620 20:13:32.076251 26118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:40919 every 8 connection(s)
I20260620 20:13:32.076483 25988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:32.078364 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 25988
I20260620 20:13:32.078382 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.065213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:32885 (local address 127.24.20.190:39683)
0620 20:13:32.065337 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.065340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.066383 (+ 1043us) server_negotiation.cc:407] Connection header received
0620 20:13:32.067335 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.067341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.067412 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.067508 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:32.068876 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.069527 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.070827 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.071048 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.071165 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.074467 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:32.074471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:32.074473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:32.074501 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:32.076033 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.076635 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.076640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.076642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.076723 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.077114 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.077117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.077119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.077331 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:32.077440 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.078051 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.078201 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:13:32.078469 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:13:32.078732 24658 external_mini_cluster.cc:1428] Setting key 943acb59d8557313309734decbdaf828
I20260620 20:13:32.079244 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.065775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:39683 (local address 127.24.20.130:32885)
0620 20:13:32.066241 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.066279 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.067090 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.067795 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.067805 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.068223 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:32.068736 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.068750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.069679 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.069683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.070705 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.070714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.071192 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.071196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.071364 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.072011 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:32.072025 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:32.074309 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:32.076145 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:32.076154 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:32.076169 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:32.076508 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:32.076821 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:32.076824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:32.076826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:32.077003 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:32.077593 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:32.077599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:32.077831 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.077978 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.078292 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":143,"threads_started":1}
I20260620 20:13:32.080506 26119 heartbeater.cc:347] Connected to a master server at 127.24.20.190:39683
I20260620 20:13:32.080672 26119 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:32.081189 26119 heartbeater.cc:511] Master 127.24.20.190:39683 requested a full tablet report, sending...
I20260620 20:13:32.082245 25775 ts_manager.cc:194] Registered new tserver with Master: 97ef923e61924e55ae2ded08b5f5ffde (127.24.20.130:40919)
I20260620 20:13:32.082903 25775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:32885
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
I20260620 20:13:32.090096 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.082998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:54177 (local address 127.24.20.190:39683)
0620 20:13:32.083138 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.083142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.083303 (+ 161us) server_negotiation.cc:407] Connection header received
0620 20:13:32.083409 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.083411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.083483 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.083558 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:32.084050 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.084713 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.085499 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.085735 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.085849 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.086901 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:32.086904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:32.086906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:32.086928 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:32.088364 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.088867 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.088872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.088874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.088965 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.089259 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.089264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.089266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.089465 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:32.089603 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.089852 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.089955 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:13:32.090106 26126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.082653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:39683 (local address 127.24.20.130:54177)
0620 20:13:32.083195 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.083211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.083322 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.083576 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.083580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.083794 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:32.083926 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.083932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.084829 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.084833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.085380 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.085388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.085847 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.085851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.085939 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.086453 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:32.086467 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:32.086769 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:32.088491 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:32.088495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:32.088497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:32.088771 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:32.089079 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:32.089082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:32.089084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:32.089168 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:32.089591 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:32.089595 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:32.089688 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.089791 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.089923 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:13:32.105401 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:39683
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:32.214485 26128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:32.214733 26128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:32.214775 26128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:32.218654 26128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:32.218711 26128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:32.218751 26128 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:13:32.218784 26128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:13:32.224452 26128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.20.190:39683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:32.225494 26128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:32.226503 26128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:13:32.235702 26128 server_base.cc:1034] running on GCE node
W20260620 20:13:32.235723 26134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:32.235615 26136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:32.235615 26133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:32.236304 26128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:32.236982 26128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:32.238134 26128 hybrid_clock.cc:648] HybridClock initialized: now 1781986412238108 us; error 38 us; skew 500 ppm
Jun 20 20:13:32 dist-test-slave-pflr krb5kdc[25731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986412, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:32.240929 26128 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:13:32.241779 26128 webserver.cc:460] Webserver started at http://127.24.20.131:45717/ using document root <none> and password file <none>
I20260620 20:13:32.242307 26128 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:32.242364 26128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:32.242575 26128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:32.244635 26128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "fc1b68f97b2043308283fec6420a276a"
format_stamp: "Formatted at 2026-06-20 20:13:32 on dist-test-slave-pflr"
server_key: "7a3204218223a92c5374d3ea3740c7c2"
server_key_iv: "86608d411e23623ca301ea60f37de76f"
server_key_version: "encryptionkey@0"
I20260620 20:13:32.245203 26128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "fc1b68f97b2043308283fec6420a276a"
format_stamp: "Formatted at 2026-06-20 20:13:32 on dist-test-slave-pflr"
server_key: "7a3204218223a92c5374d3ea3740c7c2"
server_key_iv: "86608d411e23623ca301ea60f37de76f"
server_key_version: "encryptionkey@0"
I20260620 20:13:32.248720 26128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:13:32.251042 26143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:32.252156 26128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:32.252267 26128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "fc1b68f97b2043308283fec6420a276a"
format_stamp: "Formatted at 2026-06-20 20:13:32 on dist-test-slave-pflr"
server_key: "7a3204218223a92c5374d3ea3740c7c2"
server_key_iv: "86608d411e23623ca301ea60f37de76f"
server_key_version: "encryptionkey@0"
I20260620 20:13:32.252367 26128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:32.271072 26128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:32.274288 26128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:32.274487 26128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:32.275846 26128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:32.275918 26128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:32.275992 26128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:32.276034 26128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:13:32 dist-test-slave-pflr krb5kdc[25731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986412, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:32.286988 26128 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:38797
I20260620 20:13:32.287048 26258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:38797 every 8 connection(s)
I20260620 20:13:32.287806 26128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:13:32.289948 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.276798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:47913 (local address 127.24.20.190:39683)
0620 20:13:32.276949 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.276954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.277859 (+ 905us) server_negotiation.cc:407] Connection header received
0620 20:13:32.279077 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.279081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.279146 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.279258 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:32.280486 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.281209 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.282563 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.282905 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.283032 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.286347 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:32.286351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:32.286353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:32.286382 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:32.287928 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.288493 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.288496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.288498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.288585 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.288936 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.288939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.288941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.289116 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:32.289221 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.289660 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.289785 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:13:32.290629 26152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.277174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:39683 (local address 127.24.20.131:47913)
0620 20:13:32.277687 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.277742 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.278840 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.279418 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.279425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.279864 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:32.280346 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.280359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.281375 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.281378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.282445 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.282455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.283072 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.283075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.283259 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.283833 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:32.283855 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:32.286174 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:32.288034 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:32.288039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:32.288052 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:32.288363 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:32.288695 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:32.288698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:32.288700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:32.288833 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:32.289218 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:32.289224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:32.289443 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.289618 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.289897 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":99,"threads_started":1}
I20260620 20:13:32.291230 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 26128
I20260620 20:13:32.291355 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:13:32.291720 24658 external_mini_cluster.cc:1428] Setting key 50182e0ba8098306795ef9c01d6aede8
I20260620 20:13:32.291858 26259 heartbeater.cc:347] Connected to a master server at 127.24.20.190:39683
I20260620 20:13:32.291972 26259 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:32.292469 26259 heartbeater.cc:511] Master 127.24.20.190:39683 requested a full tablet report, sending...
I20260620 20:13:32.293569 25776 ts_manager.cc:194] Registered new tserver with Master: fc1b68f97b2043308283fec6420a276a (127.24.20.131:38797)
I20260620 20:13:32.294304 25776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:47913
I20260620 20:13:32.300318 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.293992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:39683 (local address 127.24.20.131:52989)
0620 20:13:32.294280 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.294295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.294392 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.294692 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.294694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.294873 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:32.294985 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.294991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.295696 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.295699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.296421 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.296429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.296964 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.296966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.297032 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.297433 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:32.297443 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:32.297734 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:32.299184 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:32.299187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:32.299188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:32.299366 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:32.299583 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:32.299585 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:32.299586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:32.299656 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:32.299939 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:32.299941 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:32.300004 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.300078 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.300183 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20260620 20:13:32.300325 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.294075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:52989 (local address 127.24.20.190:39683)
0620 20:13:32.294187 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.294194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.294365 (+ 171us) server_negotiation.cc:407] Connection header received
0620 20:13:32.294534 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.294537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.294597 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.294671 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:32.295095 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.295571 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.296542 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.296856 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.296941 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.297863 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:32.297866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:32.297869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:32.297888 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:32.299091 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.299444 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.299446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.299447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.299512 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.299726 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.299728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.299729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.299869 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:32.299940 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.300129 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.300212 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:13:32.305219 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:13:32 dist-test-slave-pflr krb5kdc[25731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:13:32 dist-test-slave-pflr krb5kdc[25731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:32.330368 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.322667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.24.20.190:39683)
0620 20:13:32.322792 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.322795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.322806 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:13:32.322843 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.322845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.322882 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.322943 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:32.323663 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.324159 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.325014 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.325270 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.325346 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.327635 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:32.327644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:32.327646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:32.327666 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:32.329108 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.329457 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.329459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.329460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.329529 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.329743 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.329745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.329746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.329899 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:32.329996 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.330168 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.330250 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:13:32.333282 25776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:13:32.336241 25776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:32.350615 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.345008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:40919 (local address 127.0.0.1:46260)
0620 20:13:32.345469 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.345523 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.345659 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.346264 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.346270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.346292 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:32.346431 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.346440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.347695 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.347698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.349000 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.349008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.350036 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.350039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.350144 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.350212 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.350350 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.350460 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":73,"threads_started":1}
I20260620 20:13:32.350668 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.345121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46260 (local address 127.24.20.130:40919)
0620 20:13:32.345552 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.345566 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.345590 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:13:32.345885 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.345891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.346102 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.346274 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:32.346617 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.347585 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.349148 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.349734 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.349798 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.349912 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.350428 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.350523 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":131,"threads_started":1}
I20260620 20:13:32.351459 26276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.345109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:38045 (local address 127.0.0.1:33518)
0620 20:13:32.345989 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.346002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.346088 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.347289 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.347292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.347310 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:32.347396 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.347402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.348411 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.348414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.349859 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.349870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.351081 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.351084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.351178 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.351200 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.351269 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.351332 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":98,"thread_start_us":83,"threads_started":1}
I20260620 20:13:32.351897 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.346161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.24.20.129:38045)
0620 20:13:32.346779 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.346792 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.346812 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:13:32.346873 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.346877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.347038 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.347160 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:32.347518 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.348285 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.350005 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.350776 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.350878 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.351000 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.351672 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.351766 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":80,"threads_started":1}
I20260620 20:13:32.352484 26280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.345910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.24.20.131:38797)
0620 20:13:32.346555 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.346568 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.346595 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:13:32.346744 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.346749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.346938 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.347093 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:32.347296 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.348399 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.349988 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.350966 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.351080 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.351229 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.351374 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.351456 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":101,"threads_started":1}
I20260620 20:13:32.351573 26279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.345731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:38797 (local address 127.0.0.1:39002)
0620 20:13:32.346518 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.346531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.346615 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.347085 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.347088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.347105 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:32.347206 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.347212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.348509 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.348512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.349865 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.349871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.351129 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.351133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.351231 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.351259 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.351334 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.351405 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":89,"thread_start_us":92,"threads_started":1}
I20260620 20:13:32.354285 25913 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6bde2a91cc4c72addc968c6e7099b9 (DEFAULT_TABLE table=test-table [id=9dfe56498cc94dabaab34ccc30a32d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:32.354792 26182 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6bde2a91cc4c72addc968c6e7099b9 (DEFAULT_TABLE table=test-table [id=9dfe56498cc94dabaab34ccc30a32d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:32.355297 25913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6bde2a91cc4c72addc968c6e7099b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:32.355783 26182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6bde2a91cc4c72addc968c6e7099b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:32.361608 26053 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6bde2a91cc4c72addc968c6e7099b9 (DEFAULT_TABLE table=test-table [id=9dfe56498cc94dabaab34ccc30a32d80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:32.362175 26283 tablet_bootstrap.cc:492] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a: Bootstrap starting.
I20260620 20:13:32.362608 26053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f6bde2a91cc4c72addc968c6e7099b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:32.362589 26282 tablet_bootstrap.cc:492] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511: Bootstrap starting.
I20260620 20:13:32.364176 26283 tablet_bootstrap.cc:654] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:32.364934 26282 tablet_bootstrap.cc:654] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:32.364953 26283 log.cc:824] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:32.365743 26282 log.cc:824] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:32.367187 26283 tablet_bootstrap.cc:492] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a: No bootstrap required, opened a new log
I20260620 20:13:32.367391 26283 ts_tablet_manager.cc:1397] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:13:32.367722 26282 tablet_bootstrap.cc:492] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511: No bootstrap required, opened a new log
I20260620 20:13:32.367900 26282 ts_tablet_manager.cc:1397] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:13:32.369946 26286 tablet_bootstrap.cc:492] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde: Bootstrap starting.
I20260620 20:13:32.370759 26283 raft_consensus.cc:348] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.371026 26283 raft_consensus.cc:374] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:32.371107 26283 raft_consensus.cc:729] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc1b68f97b2043308283fec6420a276a, State: Initialized, Role: FOLLOWER
I20260620 20:13:32.371423 26282 raft_consensus.cc:348] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.371567 26283 consensus_queue.cc:260] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.371631 26282 raft_consensus.cc:374] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:32.371695 26282 raft_consensus.cc:729] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01402125ffe43aea3679f5dc72ff511, State: Initialized, Role: FOLLOWER
I20260620 20:13:32.372043 26286 tablet_bootstrap.cc:654] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:32.372156 26282 consensus_queue.cc:260] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.372388 26259 heartbeater.cc:503] Master 127.24.20.190:39683 was elected leader, sending a full tablet report...
I20260620 20:13:32.372550 26283 ts_tablet_manager.cc:1428] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:13:32.372846 26286 log.cc:824] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:32.374315 25979 heartbeater.cc:503] Master 127.24.20.190:39683 was elected leader, sending a full tablet report...
I20260620 20:13:32.374352 26282 ts_tablet_manager.cc:1428] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 20:13:32.374740 26286 tablet_bootstrap.cc:492] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde: No bootstrap required, opened a new log
I20260620 20:13:32.374890 26286 ts_tablet_manager.cc:1397] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:13:32.378089 26286 raft_consensus.cc:348] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.378259 26286 raft_consensus.cc:374] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:32.378340 26286 raft_consensus.cc:729] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ef923e61924e55ae2ded08b5f5ffde, State: Initialized, Role: FOLLOWER
I20260620 20:13:32.378676 26286 consensus_queue.cc:260] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.379335 26119 heartbeater.cc:503] Master 127.24.20.190:39683 was elected leader, sending a full tablet report...
I20260620 20:13:32.379370 26286 ts_tablet_manager.cc:1428] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 20:13:32.427744 26290 raft_consensus.cc:482] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:32.427965 26290 raft_consensus.cc:504] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.429216 26290 leader_election.cc:288] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e01402125ffe43aea3679f5dc72ff511 (127.24.20.129:38045), fc1b68f97b2043308283fec6420a276a (127.24.20.131:38797)
I20260620 20:13:32.433921 26280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.429807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:43587 (local address 127.24.20.131:38797)
0620 20:13:32.429937 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.429940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.430123 (+ 183us) server_negotiation.cc:407] Connection header received
0620 20:13:32.430334 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.430340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.430396 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.430493 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:32.430746 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.431266 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.432994 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.433387 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.433513 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.433569 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.433737 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.433803 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:13:32.433949 26291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.429687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:38797 (local address 127.24.20.130:43587)
0620 20:13:32.430016 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.430031 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.430220 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.430502 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.430504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.430519 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:32.430645 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.430649 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.431368 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.431374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.432880 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.432889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.433503 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.433507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.433608 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.433629 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.433724 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.433791 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":21,"spinlock_wait_cycles":22784,"thread_start_us":95,"threads_started":1}
I20260620 20:13:32.434721 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.429411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:38045 (local address 127.24.20.130:34639)
0620 20:13:32.429544 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.429556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.429681 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.430157 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.430161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.430176 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:32.430311 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.430317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.431472 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.431476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.433001 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.433012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.434215 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.434220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.434369 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.434394 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.434473 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.434562 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"spinlock_wait_cycles":1920}
I20260620 20:13:32.434736 26213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6bde2a91cc4c72addc968c6e7099b9" candidate_uuid: "97ef923e61924e55ae2ded08b5f5ffde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc1b68f97b2043308283fec6420a276a" is_pre_election: true
I20260620 20:13:32.435055 26213 raft_consensus.cc:2454] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ef923e61924e55ae2ded08b5f5ffde in term 0.
I20260620 20:13:32.435245 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.429490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:34639 (local address 127.24.20.129:38045)
0620 20:13:32.429627 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.429632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.429657 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:13:32.430003 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.430005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.430047 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.430141 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:32.430414 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.431319 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.433189 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.433925 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.434065 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.434146 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.434991 (+ 845us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.435089 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:13:32.435786 25933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6bde2a91cc4c72addc968c6e7099b9" candidate_uuid: "97ef923e61924e55ae2ded08b5f5ffde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01402125ffe43aea3679f5dc72ff511" is_pre_election: true
I20260620 20:13:32.435897 26007 leader_election.cc:302] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97ef923e61924e55ae2ded08b5f5ffde, fc1b68f97b2043308283fec6420a276a; no voters:
I20260620 20:13:32.436076 25933 raft_consensus.cc:2454] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ef923e61924e55ae2ded08b5f5ffde in term 0.
I20260620 20:13:32.436268 26290 raft_consensus.cc:2781] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:32.436383 26290 raft_consensus.cc:482] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:32.436442 26290 raft_consensus.cc:3037] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:32.437879 26290 raft_consensus.cc:504] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.438247 26290 leader_election.cc:288] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [CANDIDATE]: Term 1 election: Requested vote from peers e01402125ffe43aea3679f5dc72ff511 (127.24.20.129:38045), fc1b68f97b2043308283fec6420a276a (127.24.20.131:38797)
I20260620 20:13:32.438627 26213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6bde2a91cc4c72addc968c6e7099b9" candidate_uuid: "97ef923e61924e55ae2ded08b5f5ffde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc1b68f97b2043308283fec6420a276a"
I20260620 20:13:32.438751 26213 raft_consensus.cc:3037] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:32.440315 26213 raft_consensus.cc:2454] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ef923e61924e55ae2ded08b5f5ffde in term 1.
I20260620 20:13:32.440618 25933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f6bde2a91cc4c72addc968c6e7099b9" candidate_uuid: "97ef923e61924e55ae2ded08b5f5ffde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01402125ffe43aea3679f5dc72ff511"
I20260620 20:13:32.440627 26007 leader_election.cc:302] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97ef923e61924e55ae2ded08b5f5ffde, fc1b68f97b2043308283fec6420a276a; no voters:
I20260620 20:13:32.440759 25933 raft_consensus.cc:3037] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:32.440836 26290 raft_consensus.cc:2781] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:32.441053 26290 raft_consensus.cc:686] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [term 1 LEADER]: Becoming Leader. State: Replica: 97ef923e61924e55ae2ded08b5f5ffde, State: Running, Role: LEADER
I20260620 20:13:32.441500 26290 consensus_queue.cc:237] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.442035 25933 raft_consensus.cc:2454] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ef923e61924e55ae2ded08b5f5ffde in term 1.
I20260620 20:13:32.445346 25775 catalog_manager.cc:5495] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde reported cstate change: term changed from 0 to 1, leader changed from <none> to 97ef923e61924e55ae2ded08b5f5ffde (127.24.20.130). New cstate: current_term: 1 leader_uuid: "97ef923e61924e55ae2ded08b5f5ffde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } health_report { overall_health: UNKNOWN } } }
I20260620 20:13:32.475606 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.472439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46274 (local address 127.24.20.130:40919)
0620 20:13:32.472559 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.472563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.472574 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:13:32.472747 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.472749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.472800 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.472882 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:32.473108 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.473584 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.474381 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.474585 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.474666 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.474918 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:32.475279 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:32.475358 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.475440 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.475485 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:13:32.480894 26213 raft_consensus.cc:1261] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a [term 1 FOLLOWER]: Refusing update from remote peer 97ef923e61924e55ae2ded08b5f5ffde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:32.480893 25933 raft_consensus.cc:1261] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511 [term 1 FOLLOWER]: Refusing update from remote peer 97ef923e61924e55ae2ded08b5f5ffde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:32.481541 26290 consensus_queue.cc:1035] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:32.481717 26292 consensus_queue.cc:1035] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:32.491438 26296 mvcc.cc:205] Tried to move back new op lower bound from 7299016345514819584 to 7299016345364901888. Current Snapshot: MvccSnapshot[applied={T|T < 7299016345514819584}]
I20260620 20:13:32.500313 25775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:32.500515 25775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:32.503022 25775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8: Sending DeleteTablet for 3 replicas of tablet 9f6bde2a91cc4c72addc968c6e7099b9
I20260620 20:13:32.503875 25913 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f6bde2a91cc4c72addc968c6e7099b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:32 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:33518
I20260620 20:13:32.503902 26182 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f6bde2a91cc4c72addc968c6e7099b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:32 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:39002
I20260620 20:13:32.504436 26053 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f6bde2a91cc4c72addc968c6e7099b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:32 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:46260
I20260620 20:13:32.504549 26305 tablet_replica.cc:330] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511: stopping tablet replica
I20260620 20:13:32.504876 26307 tablet_replica.cc:330] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde: stopping tablet replica
I20260620 20:13:32.504823 25775 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 20:13:32.504952 26306 tablet_replica.cc:330] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a: stopping tablet replica
W20260620 20:13:32.505203 25775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:32.505334 26307 raft_consensus.cc:2229] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:13:32.505411 26306 raft_consensus.cc:2229] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:32.505712 26307 raft_consensus.cc:2258] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:32.505728 26306 raft_consensus.cc:2258] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:32.506371 26305 raft_consensus.cc:2229] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:32.507215 26307 ts_tablet_manager.cc:1905] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:32.507591 26305 raft_consensus.cc:2258] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:32.508195 26306 ts_tablet_manager.cc:1905] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:32.510293 26307 ts_tablet_manager.cc:1918] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:32.510632 25913 tablet_service.cc:1467] Processing CreateTablet for tablet 5545b1efa49b45b7a00eae1b9f5e3fb1 (DEFAULT_TABLE table=test-table [id=15b3eec58be14ee58fbfb557e0aa8257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:32.510845 26053 tablet_service.cc:1467] Processing CreateTablet for tablet 5545b1efa49b45b7a00eae1b9f5e3fb1 (DEFAULT_TABLE table=test-table [id=15b3eec58be14ee58fbfb557e0aa8257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:32.510915 25913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5545b1efa49b45b7a00eae1b9f5e3fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:32.511080 26053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5545b1efa49b45b7a00eae1b9f5e3fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:32.512971 26282 tablet_bootstrap.cc:492] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511: Bootstrap starting.
I20260620 20:13:32.513342 26286 tablet_bootstrap.cc:492] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde: Bootstrap starting.
I20260620 20:13:32.510455 26307 log.cc:1196] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/9f6bde2a91cc4c72addc968c6e7099b9
I20260620 20:13:32.513616 26182 tablet_service.cc:1467] Processing CreateTablet for tablet 5545b1efa49b45b7a00eae1b9f5e3fb1 (DEFAULT_TABLE table=test-table [id=15b3eec58be14ee58fbfb557e0aa8257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:32.513857 26182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5545b1efa49b45b7a00eae1b9f5e3fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:32.513979 26282 tablet_bootstrap.cc:654] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:32.514003 26307 ts_tablet_manager.cc:1939] T 9f6bde2a91cc4c72addc968c6e7099b9 P 97ef923e61924e55ae2ded08b5f5ffde: Deleting consensus metadata
I20260620 20:13:32.514464 26286 tablet_bootstrap.cc:654] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:32.515300 25762 catalog_manager.cc:4841] TS 97ef923e61924e55ae2ded08b5f5ffde (127.24.20.130:40919): tablet 9f6bde2a91cc4c72addc968c6e7099b9 (table test-table [id=9dfe56498cc94dabaab34ccc30a32d80]) successfully deleted
I20260620 20:13:32.515743 26286 tablet_bootstrap.cc:492] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde: No bootstrap required, opened a new log
I20260620 20:13:32.515813 26286 ts_tablet_manager.cc:1397] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:13:32.516251 26283 tablet_bootstrap.cc:492] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a: Bootstrap starting.
I20260620 20:13:32.516430 26286 raft_consensus.cc:348] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.516541 26286 raft_consensus.cc:374] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:32.516578 26286 raft_consensus.cc:729] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ef923e61924e55ae2ded08b5f5ffde, State: Initialized, Role: FOLLOWER
I20260620 20:13:32.516768 26286 consensus_queue.cc:260] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.517103 26306 ts_tablet_manager.cc:1918] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:32.517207 26306 log.cc:1196] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/9f6bde2a91cc4c72addc968c6e7099b9
I20260620 20:13:32.517345 26283 tablet_bootstrap.cc:654] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:32.517542 26306 ts_tablet_manager.cc:1939] T 9f6bde2a91cc4c72addc968c6e7099b9 P fc1b68f97b2043308283fec6420a276a: Deleting consensus metadata
I20260620 20:13:32.517551 26286 ts_tablet_manager.cc:1428] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 20:13:32.518747 25763 catalog_manager.cc:4841] TS fc1b68f97b2043308283fec6420a276a (127.24.20.131:38797): tablet 9f6bde2a91cc4c72addc968c6e7099b9 (table test-table [id=9dfe56498cc94dabaab34ccc30a32d80]) successfully deleted
I20260620 20:13:32.518996 26283 tablet_bootstrap.cc:492] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a: No bootstrap required, opened a new log
I20260620 20:13:32.519089 26283 ts_tablet_manager.cc:1397] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 20:13:32.519124 26305 ts_tablet_manager.cc:1905] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:32.519558 26283 raft_consensus.cc:348] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.519694 26283 raft_consensus.cc:374] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:32.519738 26283 raft_consensus.cc:729] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc1b68f97b2043308283fec6420a276a, State: Initialized, Role: FOLLOWER
I20260620 20:13:32.519795 26282 tablet_bootstrap.cc:492] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511: No bootstrap required, opened a new log
I20260620 20:13:32.519878 26282 ts_tablet_manager.cc:1397] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 20:13:32.519883 26283 consensus_queue.cc:260] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.520128 26283 ts_tablet_manager.cc:1428] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 20:13:32.520339 26282 raft_consensus.cc:348] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.520439 26282 raft_consensus.cc:374] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:32.520470 26282 raft_consensus.cc:729] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01402125ffe43aea3679f5dc72ff511, State: Initialized, Role: FOLLOWER
I20260620 20:13:32.520608 26282 consensus_queue.cc:260] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.521098 26282 ts_tablet_manager.cc:1428] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:32.522748 26305 ts_tablet_manager.cc:1918] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:32.522903 26305 log.cc:1196] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/wal/wals/9f6bde2a91cc4c72addc968c6e7099b9
I20260620 20:13:32.523312 26305 ts_tablet_manager.cc:1939] T 9f6bde2a91cc4c72addc968c6e7099b9 P e01402125ffe43aea3679f5dc72ff511: Deleting consensus metadata
I20260620 20:13:32.524533 25763 catalog_manager.cc:4841] TS e01402125ffe43aea3679f5dc72ff511 (127.24.20.129:38045): tablet 9f6bde2a91cc4c72addc968c6e7099b9 (table test-table [id=9dfe56498cc94dabaab34ccc30a32d80]) successfully deleted
I20260620 20:13:32.529181 26300 raft_consensus.cc:482] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:32.529310 26300 raft_consensus.cc:504] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.529719 26300 leader_election.cc:288] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e01402125ffe43aea3679f5dc72ff511 (127.24.20.129:38045), fc1b68f97b2043308283fec6420a276a (127.24.20.131:38797)
I20260620 20:13:32.530121 25933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5545b1efa49b45b7a00eae1b9f5e3fb1" candidate_uuid: "97ef923e61924e55ae2ded08b5f5ffde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01402125ffe43aea3679f5dc72ff511" is_pre_election: true
I20260620 20:13:32.530107 26213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5545b1efa49b45b7a00eae1b9f5e3fb1" candidate_uuid: "97ef923e61924e55ae2ded08b5f5ffde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc1b68f97b2043308283fec6420a276a" is_pre_election: true
I20260620 20:13:32.530262 26213 raft_consensus.cc:2454] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ef923e61924e55ae2ded08b5f5ffde in term 0.
I20260620 20:13:32.530268 25933 raft_consensus.cc:2454] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97ef923e61924e55ae2ded08b5f5ffde in term 0.
I20260620 20:13:32.530592 26007 leader_election.cc:302] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97ef923e61924e55ae2ded08b5f5ffde, e01402125ffe43aea3679f5dc72ff511; no voters:
I20260620 20:13:32.530753 26300 raft_consensus.cc:2781] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:32.530825 26300 raft_consensus.cc:482] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:32.530884 26300 raft_consensus.cc:3037] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:32.531781 26300 raft_consensus.cc:504] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.532169 26300 leader_election.cc:288] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [CANDIDATE]: Term 1 election: Requested vote from peers e01402125ffe43aea3679f5dc72ff511 (127.24.20.129:38045), fc1b68f97b2043308283fec6420a276a (127.24.20.131:38797)
I20260620 20:13:32.532486 25933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5545b1efa49b45b7a00eae1b9f5e3fb1" candidate_uuid: "97ef923e61924e55ae2ded08b5f5ffde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01402125ffe43aea3679f5dc72ff511"
I20260620 20:13:32.532639 25933 raft_consensus.cc:3037] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:32.532693 26213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5545b1efa49b45b7a00eae1b9f5e3fb1" candidate_uuid: "97ef923e61924e55ae2ded08b5f5ffde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc1b68f97b2043308283fec6420a276a"
I20260620 20:13:32.532793 26213 raft_consensus.cc:3037] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:32.533416 25933 raft_consensus.cc:2454] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ef923e61924e55ae2ded08b5f5ffde in term 1.
I20260620 20:13:32.533766 26213 raft_consensus.cc:2454] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97ef923e61924e55ae2ded08b5f5ffde in term 1.
I20260620 20:13:32.533816 26007 leader_election.cc:302] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97ef923e61924e55ae2ded08b5f5ffde, e01402125ffe43aea3679f5dc72ff511; no voters:
I20260620 20:13:32.534034 26300 raft_consensus.cc:2781] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:32.534155 26300 raft_consensus.cc:686] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [term 1 LEADER]: Becoming Leader. State: Replica: 97ef923e61924e55ae2ded08b5f5ffde, State: Running, Role: LEADER
I20260620 20:13:32.534324 26300 consensus_queue.cc:237] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.535974 25775 catalog_manager.cc:5495] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde reported cstate change: term changed from 0 to 1, leader changed from <none> to 97ef923e61924e55ae2ded08b5f5ffde (127.24.20.130). New cstate: current_term: 1 leader_uuid: "97ef923e61924e55ae2ded08b5f5ffde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } health_report { overall_health: UNKNOWN } } }
W20260620 20:13:32.538811 26260 tablet.cc:2369] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:13:32 dist-test-slave-pflr krb5kdc[25731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986411, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:32.575774 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.562310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:39683 (local address 127.0.0.1:55654)
0620 20:13:32.562689 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.562702 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.562812 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.563081 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.563084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.563555 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:32.563671 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.563679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.564777 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.564780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.565670 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.565679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.566233 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.566237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.566324 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.570336 (+ 4012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:32.570352 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:32.572096 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:32.574224 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:32.574232 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:32.574238 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:32.574532 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:32.574816 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:32.574818 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:32.574819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:32.574895 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:32.575281 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:32.575286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:32.575381 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.575537 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.575648 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":122,"threads_started":1}
I20260620 20:13:32.575899 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.562415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.24.20.190:39683)
0620 20:13:32.562536 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.562541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.562788 (+ 247us) server_negotiation.cc:407] Connection header received
0620 20:13:32.562907 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.562910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.562955 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.563040 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:32.563817 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.564678 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.565788 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.566117 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.566219 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.572234 (+ 6015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:32.572238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:32.572240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:32.572261 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:32.574098 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.574633 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.574636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.574645 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.574727 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.574977 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.574980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.574982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.575185 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:32.575280 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.575609 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.575727 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 20:13:32.577659 26120 tablet.cc:2369] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:13:32.579919 25775 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:55654:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:13:32.580747 25775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:32.585474 26053 tablet_service.cc:1467] Processing CreateTablet for tablet 8db8f938946345ea8a385280d17e9f0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3d50f23c4ecc4d6da57df79d5795f978]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:13:32.585470 25913 tablet_service.cc:1467] Processing CreateTablet for tablet 8db8f938946345ea8a385280d17e9f0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3d50f23c4ecc4d6da57df79d5795f978]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:13:32.585844 26053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db8f938946345ea8a385280d17e9f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:32.585844 25913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db8f938946345ea8a385280d17e9f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:32.585741 26182 tablet_service.cc:1467] Processing CreateTablet for tablet 8db8f938946345ea8a385280d17e9f0c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3d50f23c4ecc4d6da57df79d5795f978]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:13:32.586071 26182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db8f938946345ea8a385280d17e9f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:32.588657 26282 tablet_bootstrap.cc:492] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511: Bootstrap starting.
I20260620 20:13:32.588811 26283 tablet_bootstrap.cc:492] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a: Bootstrap starting.
I20260620 20:13:32.588828 26286 tablet_bootstrap.cc:492] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde: Bootstrap starting.
I20260620 20:13:32.589716 26286 tablet_bootstrap.cc:654] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:32.589716 26282 tablet_bootstrap.cc:654] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:32.589718 26283 tablet_bootstrap.cc:654] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:32.591297 26283 tablet_bootstrap.cc:492] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a: No bootstrap required, opened a new log
I20260620 20:13:32.591375 26283 ts_tablet_manager.cc:1397] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 20:13:32.591785 26283 raft_consensus.cc:348] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.591912 26283 raft_consensus.cc:374] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:32.591945 26283 raft_consensus.cc:729] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc1b68f97b2043308283fec6420a276a, State: Initialized, Role: FOLLOWER
I20260620 20:13:32.592056 26283 consensus_queue.cc:260] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.592293 26283 ts_tablet_manager.cc:1428] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:32.592681 26287 tablet_replica.cc:439] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } } }
I20260620 20:13:32.592828 26287 tablet_replica.cc:442] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:13:32.593739 26286 tablet_bootstrap.cc:492] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde: No bootstrap required, opened a new log
I20260620 20:13:32.593797 26282 tablet_bootstrap.cc:492] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511: No bootstrap required, opened a new log
I20260620 20:13:32.593820 26286 ts_tablet_manager.cc:1397] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260620 20:13:32.593865 26282 ts_tablet_manager.cc:1397] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 20:13:32.594245 26282 raft_consensus.cc:348] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.594341 26282 raft_consensus.cc:374] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:32.594272 26286 raft_consensus.cc:348] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.594372 26282 raft_consensus.cc:729] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01402125ffe43aea3679f5dc72ff511, State: Initialized, Role: FOLLOWER
I20260620 20:13:32.594398 26286 raft_consensus.cc:374] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:32.594436 26286 raft_consensus.cc:729] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ef923e61924e55ae2ded08b5f5ffde, State: Initialized, Role: FOLLOWER
I20260620 20:13:32.594480 26282 consensus_queue.cc:260] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.594574 26286 consensus_queue.cc:260] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.594723 26282 ts_tablet_manager.cc:1428] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:32.594837 26286 ts_tablet_manager.cc:1428] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 20:13:32.594913 26289 tablet_replica.cc:439] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } } }
I20260620 20:13:32.595031 26289 tablet_replica.cc:442] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:13:32.595078 26290 tablet_replica.cc:439] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } } }
I20260620 20:13:32.595189 26290 tablet_replica.cc:442] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 20:13:32.604879 25980 tablet.cc:2369] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:13:32.638430 26287 raft_consensus.cc:482] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:32.638579 26287 raft_consensus.cc:504] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.639883 26287 leader_election.cc:288] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e01402125ffe43aea3679f5dc72ff511 (127.24.20.129:38045), 97ef923e61924e55ae2ded08b5f5ffde (127.24.20.130:40919)
I20260620 20:13:32.644766 26152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.640069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:38045 (local address 127.24.20.131:57125)
0620 20:13:32.640256 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.640275 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.640366 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.640895 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.640898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.640914 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:32.641028 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.641035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.642089 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.642092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.643370 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.643380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.644310 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.644314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.644417 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.644443 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.644525 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.644620 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 20:13:32.644781 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.640473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:57125 (local address 127.24.20.129:38045)
0620 20:13:32.640658 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.640664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.640681 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:13:32.640734 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.640740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.640792 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.640888 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:32.641150 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.641964 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.643539 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.644184 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.644313 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.644368 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.644552 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.644657 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:13:32.645175 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.640248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:40505 (local address 127.24.20.130:40919)
0620 20:13:32.640372 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.640378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.640679 (+ 301us) server_negotiation.cc:407] Connection header received
0620 20:13:32.640821 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.640824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.640876 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.640961 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:32.641866 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.642639 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.643994 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.644486 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.644585 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.644650 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.644946 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.645030 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:13:32.645175 25933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db8f938946345ea8a385280d17e9f0c" candidate_uuid: "fc1b68f97b2043308283fec6420a276a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01402125ffe43aea3679f5dc72ff511" is_pre_election: true
I20260620 20:13:32.645350 25933 raft_consensus.cc:2454] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc1b68f97b2043308283fec6420a276a in term 0.
I20260620 20:13:32.645543 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.640069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:40919 (local address 127.24.20.131:40505)
0620 20:13:32.640578 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.640591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.640711 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.641014 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.641016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.641029 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:32.641728 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.641741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.642775 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.642778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.643880 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.643888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.644703 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.644707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.644797 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.644823 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.645298 (+ 475us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.645383 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":136,"thread_start_us":101,"threads_started":1}
I20260620 20:13:32.645785 26147 leader_election.cc:302] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e01402125ffe43aea3679f5dc72ff511, fc1b68f97b2043308283fec6420a276a; no voters:
I20260620 20:13:32.646054 26287 raft_consensus.cc:2781] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:32.646108 26287 raft_consensus.cc:482] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:32.646163 26287 raft_consensus.cc:3037] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:32.646314 26073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db8f938946345ea8a385280d17e9f0c" candidate_uuid: "fc1b68f97b2043308283fec6420a276a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ef923e61924e55ae2ded08b5f5ffde" is_pre_election: true
I20260620 20:13:32.646528 26073 raft_consensus.cc:2454] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc1b68f97b2043308283fec6420a276a in term 0.
I20260620 20:13:32.647239 26287 raft_consensus.cc:504] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.647574 26287 leader_election.cc:288] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [CANDIDATE]: Term 1 election: Requested vote from peers e01402125ffe43aea3679f5dc72ff511 (127.24.20.129:38045), 97ef923e61924e55ae2ded08b5f5ffde (127.24.20.130:40919)
I20260620 20:13:32.647855 25933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db8f938946345ea8a385280d17e9f0c" candidate_uuid: "fc1b68f97b2043308283fec6420a276a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01402125ffe43aea3679f5dc72ff511"
I20260620 20:13:32.647890 26073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db8f938946345ea8a385280d17e9f0c" candidate_uuid: "fc1b68f97b2043308283fec6420a276a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97ef923e61924e55ae2ded08b5f5ffde"
I20260620 20:13:32.647974 25933 raft_consensus.cc:3037] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:32.648007 26073 raft_consensus.cc:3037] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:32.649003 26073 raft_consensus.cc:2454] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc1b68f97b2043308283fec6420a276a in term 1.
I20260620 20:13:32.649150 25933 raft_consensus.cc:2454] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc1b68f97b2043308283fec6420a276a in term 1.
I20260620 20:13:32.649269 26146 leader_election.cc:302] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97ef923e61924e55ae2ded08b5f5ffde, fc1b68f97b2043308283fec6420a276a; no voters:
I20260620 20:13:32.649453 26287 raft_consensus.cc:2781] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:32.649667 26287 raft_consensus.cc:686] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [term 1 LEADER]: Becoming Leader. State: Replica: fc1b68f97b2043308283fec6420a276a, State: Running, Role: LEADER
I20260620 20:13:32.649919 26287 consensus_queue.cc:237] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } }
I20260620 20:13:32.651196 26322 tablet_replica.cc:439] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a: TxnStatusTablet state changed. Reason: New leader fc1b68f97b2043308283fec6420a276a. Latest consensus state: current_term: 1 leader_uuid: "fc1b68f97b2043308283fec6420a276a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } } }
I20260620 20:13:32.651317 26322 tablet_replica.cc:442] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:13:32.651654 26324 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:13:32.651796 26324 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:13:32.652385 25776 catalog_manager.cc:5495] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a reported cstate change: term changed from 0 to 1, leader changed from <none> to fc1b68f97b2043308283fec6420a276a (127.24.20.131). New cstate: current_term: 1 leader_uuid: "fc1b68f97b2043308283fec6420a276a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } health_report { overall_health: HEALTHY } } }
I20260620 20:13:32.678861 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.675156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:38797 (local address 127.0.0.1:39014)
0620 20:13:32.675382 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.675396 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.675527 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.675846 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.675849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.675856 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:13:32.675956 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.675962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.676707 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.676711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.677322 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.677330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.677773 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.677776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.677861 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.677915 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:13:32.678577 (+ 662us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:13:32.678592 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.678667 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.678747 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 20:13:32.678861 26280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.675250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.24.20.131:38797)
0620 20:13:32.675360 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.675365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.675465 (+ 100us) server_negotiation.cc:407] Connection header received
0620 20:13:32.675665 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.675668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.675730 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.675823 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:32.676066 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.676563 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.677441 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.677662 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.677764 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.678086 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:32.678473 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:32.678592 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.678676 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.678723 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:13:32.681524 25933 raft_consensus.cc:1261] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511 [term 1 FOLLOWER]: Refusing update from remote peer fc1b68f97b2043308283fec6420a276a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:32.681677 26073 raft_consensus.cc:1261] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde [term 1 FOLLOWER]: Refusing update from remote peer fc1b68f97b2043308283fec6420a276a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:32.682088 26287 consensus_queue.cc:1035] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:32.682502 26287 consensus_queue.cc:1035] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:32.683699 26289 tablet_replica.cc:439] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511: TxnStatusTablet state changed. Reason: New leader fc1b68f97b2043308283fec6420a276a. Latest consensus state: current_term: 1 leader_uuid: "fc1b68f97b2043308283fec6420a276a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } } }
I20260620 20:13:32.683835 26289 tablet_replica.cc:442] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:13:32.684094 26290 tablet_replica.cc:439] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde: TxnStatusTablet state changed. Reason: New leader fc1b68f97b2043308283fec6420a276a. Latest consensus state: current_term: 1 leader_uuid: "fc1b68f97b2043308283fec6420a276a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } } }
I20260620 20:13:32.684199 26290 tablet_replica.cc:442] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:13:32.686681 26287 tablet_replica.cc:439] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc1b68f97b2043308283fec6420a276a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } } }
I20260620 20:13:32.686796 26287 tablet_replica.cc:442] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:13:32.687366 26290 tablet_replica.cc:439] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc1b68f97b2043308283fec6420a276a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } } }
I20260620 20:13:32.687463 26290 tablet_replica.cc:442] T 8db8f938946345ea8a385280d17e9f0c P 97ef923e61924e55ae2ded08b5f5ffde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:13:32.690868 26287 tablet_replica.cc:439] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc1b68f97b2043308283fec6420a276a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } } }
I20260620 20:13:32.690969 26287 tablet_replica.cc:442] T 8db8f938946345ea8a385280d17e9f0c P fc1b68f97b2043308283fec6420a276a: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:13:32.691421 26289 tablet_replica.cc:439] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc1b68f97b2043308283fec6420a276a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 } } peers { permanent_uuid: "97ef923e61924e55ae2ded08b5f5ffde" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40919 } } peers { permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 } } }
I20260620 20:13:32.691526 26289 tablet_replica.cc:442] T 8db8f938946345ea8a385280d17e9f0c P e01402125ffe43aea3679f5dc72ff511: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:13:32.703414 26280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.700065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:43377 (local address 127.24.20.131:38797)
0620 20:13:32.700198 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.700201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.700417 (+ 216us) server_negotiation.cc:407] Connection header received
0620 20:13:32.700570 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.700573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.700698 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.700775 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:32.701015 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.701510 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.702362 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.702584 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.702698 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.702959 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:32.703032 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:32.703110 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.703234 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.703281 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:13:32.703431 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.699979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:38797 (local address 127.24.20.130:43377)
0620 20:13:32.700333 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.700344 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.700471 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.700796 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.700799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.700807 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:13:32.700913 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.700918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.701611 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.701617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.702237 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.702245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.702697 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.702701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.702792 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.702831 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:13:32.703123 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:13:32.703138 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.703222 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.703306 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":82,"threads_started":1}
I20260620 20:13:32.709298 26213 raft_consensus.cc:1261] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a [term 1 FOLLOWER]: Refusing update from remote peer 97ef923e61924e55ae2ded08b5f5ffde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:32.709328 25933 raft_consensus.cc:1261] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511 [term 1 FOLLOWER]: Refusing update from remote peer 97ef923e61924e55ae2ded08b5f5ffde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:32.709724 26300 consensus_queue.cc:1035] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01402125ffe43aea3679f5dc72ff511" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:32.709867 26290 consensus_queue.cc:1035] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc1b68f97b2043308283fec6420a276a" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:32.713788 26295 mvcc.cc:205] Tried to move back new op lower bound from 7299016346453995520 to 7299016345742090240. Current Snapshot: MvccSnapshot[applied={T|T < 7299016346453995520}]
I20260620 20:13:32.714550 26297 mvcc.cc:205] Tried to move back new op lower bound from 7299016346453995520 to 7299016345742090240. Current Snapshot: MvccSnapshot[applied={T|T < 7299016346453995520}]
I20260620 20:13:32.729787 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.726290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:36429 (local address 127.24.20.130:40919)
0620 20:13:32.726422 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.726428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.726441 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:13:32.726558 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.726561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.726601 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.726692 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:32.726932 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.727641 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.728870 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.729075 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.729150 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.729443 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:32.729501 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:32.729547 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.729651 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.729690 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:13:32.729849 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.726230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:40919 (local address 127.24.20.131:36429)
0620 20:13:32.726377 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:32.726392 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:32.726476 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:32.726709 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:32.726712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:32.726719 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:13:32.726825 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.726837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.727746 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.727749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.728758 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:32.728769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.729180 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.729184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:32.729281 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.729330 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:13:32.729576 (+ 246us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:13:32.729582 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:13:32.729649 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 20:13:32.729724 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 20:13:32.751842 25776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:32.751999 25776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:32.753849 25776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f115ded05ac4194be4a93d4051765f8: Sending DeleteTablet for 3 replicas of tablet 5545b1efa49b45b7a00eae1b9f5e3fb1
I20260620 20:13:32.754369 25913 tablet_service.cc:1514] Processing DeleteTablet for tablet 5545b1efa49b45b7a00eae1b9f5e3fb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:32 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:33518
I20260620 20:13:32.754361 26053 tablet_service.cc:1514] Processing DeleteTablet for tablet 5545b1efa49b45b7a00eae1b9f5e3fb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:32 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:46260
I20260620 20:13:32.754525 26307 tablet_replica.cc:330] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde: stopping tablet replica
I20260620 20:13:32.754524 26305 tablet_replica.cc:330] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511: stopping tablet replica
I20260620 20:13:32.754629 26307 raft_consensus.cc:2229] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:13:32.754648 26305 raft_consensus.cc:2229] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:32.754745 26305 raft_consensus.cc:2258] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:32.754806 26307 raft_consensus.cc:2258] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:32.754907 26182 tablet_service.cc:1514] Processing DeleteTablet for tablet 5545b1efa49b45b7a00eae1b9f5e3fb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:32 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:39002
I20260620 20:13:32.755065 26306 tablet_replica.cc:330] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a: stopping tablet replica
I20260620 20:13:32.755174 26306 raft_consensus.cc:2229] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:32.755285 26306 raft_consensus.cc:2258] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:32.755656 26307 ts_tablet_manager.cc:1905] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:32.756769 26305 ts_tablet_manager.cc:1905] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:32.758908 26307 ts_tablet_manager.cc:1918] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:13:32.758940 26306 ts_tablet_manager.cc:1905] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:32.758982 26307 log.cc:1196] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/5545b1efa49b45b7a00eae1b9f5e3fb1
I20260620 20:13:32.759254 26307 ts_tablet_manager.cc:1939] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P 97ef923e61924e55ae2ded08b5f5ffde: Deleting consensus metadata
I20260620 20:13:32.759915 25762 catalog_manager.cc:4841] TS 97ef923e61924e55ae2ded08b5f5ffde (127.24.20.130:40919): tablet 5545b1efa49b45b7a00eae1b9f5e3fb1 (table test-table [id=15b3eec58be14ee58fbfb557e0aa8257]) successfully deleted
I20260620 20:13:32.760005 26305 ts_tablet_manager.cc:1918] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:13:32.760068 26305 log.cc:1196] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-0/wal/wals/5545b1efa49b45b7a00eae1b9f5e3fb1
I20260620 20:13:32.760320 26305 ts_tablet_manager.cc:1939] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P e01402125ffe43aea3679f5dc72ff511: Deleting consensus metadata
I20260620 20:13:32.760974 25763 catalog_manager.cc:4841] TS e01402125ffe43aea3679f5dc72ff511 (127.24.20.129:38045): tablet 5545b1efa49b45b7a00eae1b9f5e3fb1 (table test-table [id=15b3eec58be14ee58fbfb557e0aa8257]) successfully deleted
I20260620 20:13:32.761651 26306 ts_tablet_manager.cc:1918] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:13:32.761713 26306 log.cc:1196] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/5545b1efa49b45b7a00eae1b9f5e3fb1
I20260620 20:13:32.761955 26306 ts_tablet_manager.cc:1939] T 5545b1efa49b45b7a00eae1b9f5e3fb1 P fc1b68f97b2043308283fec6420a276a: Deleting consensus metadata
I20260620 20:13:32.762578 25763 catalog_manager.cc:4841] TS fc1b68f97b2043308283fec6420a276a (127.24.20.131:38797): tablet 5545b1efa49b45b7a00eae1b9f5e3fb1 (table test-table [id=15b3eec58be14ee58fbfb557e0aa8257]) successfully deleted
Jun 20 20:13:32 dist-test-slave-pflr krb5kdc[25731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.20.129@KRBTEST.COM
I20260620 20:13:32.770556 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.759033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.24.20.129:38045)
0620 20:13:32.759160 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.759167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.760365 (+ 1198us) server_negotiation.cc:407] Connection header received
0620 20:13:32.760566 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.760569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.760654 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.760707 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:32.761612 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.762307 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.763276 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.763499 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.763605 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.766997 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:32.767007 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:32.767025 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:32.767052 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:32.768936 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.769330 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.769336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.769338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.769438 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.769680 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.769683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.769685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.769964 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:32.770165 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.770287 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.770400 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 20:13:32.771029 25973 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33534
I20260620 20:13:32.779714 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:32.772819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.24.20.190:39683)
0620 20:13:32.772945 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:32.772952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:32.772969 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:13:32.773108 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:32.773111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:32.773163 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:32.773247 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:32.773837 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.774350 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.775109 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:32.775331 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:32.775436 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:32.776807 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:32.776811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:32.776813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:32.776845 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:32.778152 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.778543 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.778548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.778550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.778635 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:32.778898 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:32.778903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:32.778905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:32.779076 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:32.779208 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:32.779431 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 20:13:32.779547 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 20:13:32.780126 25776 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55668
I20260620 20:13:32.781431 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 25848
I20260620 20:13:32.790663 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 25988
I20260620 20:13:32.797866 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 26128
I20260620 20:13:32.802953 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 25744
2026-06-20T20:13:32Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2560 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:13:32 dist-test-slave-pflr krb5kdc[26343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:13:32 dist-test-slave-pflr krb5kdc[26343](info): set up 2 sockets
Jun 20 20:13:32 dist-test-slave-pflr krb5kdc[26343](info): commencing operation
krb5kdc: starting...
W20260620 20:13:33.861220 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:13:33 dist-test-slave-pflr krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:13:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:13:33.979106 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:33721
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:33721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:34.090495 26356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:34.090905 26356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:34.090992 26356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:34.097533 26356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:34.097616 26356 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:13:34.097646 26356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:34.097688 26356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:34.097723 26356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:34.097757 26356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:34.105293 26356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:33721
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:33721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:34.106382 26356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:34.107417 26356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:34.116096 26362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:34.116070 26364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:34.116142 26356 server_base.cc:1034] running on GCE node
W20260620 20:13:34.116079 26361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:34.116964 26356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:34.118006 26356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:34.119194 26356 hybrid_clock.cc:648] HybridClock initialized: now 1781986414119168 us; error 41 us; skew 500 ppm
Jun 20 20:13:34 dist-test-slave-pflr krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986414, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:34.121877 26356 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:13:34.122741 26356 webserver.cc:460] Webserver started at http://127.24.20.190:36755/ using document root <none> and password file <none>
I20260620 20:13:34.123284 26356 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:34.123371 26356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:34.123595 26356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:34.125663 26356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "35cc233f12ab4f2d977c09d8abf5c9fb"
format_stamp: "Formatted at 2026-06-20 20:13:34 on dist-test-slave-pflr"
server_key: "f01c6269400618d90837e5c5dd366189"
server_key_iv: "ec36bf7eba3f008ba11aa0632aa549a4"
server_key_version: "encryptionkey@0"
I20260620 20:13:34.126212 26356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "35cc233f12ab4f2d977c09d8abf5c9fb"
format_stamp: "Formatted at 2026-06-20 20:13:34 on dist-test-slave-pflr"
server_key: "f01c6269400618d90837e5c5dd366189"
server_key_iv: "ec36bf7eba3f008ba11aa0632aa549a4"
server_key_version: "encryptionkey@0"
I20260620 20:13:34.129702 26356 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:13:34.131922 26371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:34.133144 26356 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:34.133277 26356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "35cc233f12ab4f2d977c09d8abf5c9fb"
format_stamp: "Formatted at 2026-06-20 20:13:34 on dist-test-slave-pflr"
server_key: "f01c6269400618d90837e5c5dd366189"
server_key_iv: "ec36bf7eba3f008ba11aa0632aa549a4"
server_key_version: "encryptionkey@0"
I20260620 20:13:34.133381 26356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:34.145066 26356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:34.147548 26356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:34.147740 26356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:34.155145 26356 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:33721
I20260620 20:13:34.155165 26433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:33721 every 8 connection(s)
I20260620 20:13:34.156025 26356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:34.158805 26434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:34.164127 26434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb: Bootstrap starting.
I20260620 20:13:34.164625 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 26356
I20260620 20:13:34.164738 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:34.164994 24658 external_mini_cluster.cc:1428] Setting key da3648436a2c32f3221dcfeff71c4ba3
I20260620 20:13:34.166823 26434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:34.167526 26434 log.cc:824] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:34.169643 26434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb: No bootstrap required, opened a new log
Jun 20 20:13:34 dist-test-slave-pflr krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:34.174140 26434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35cc233f12ab4f2d977c09d8abf5c9fb" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33721 } }
I20260620 20:13:34.174409 26434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:34.174482 26434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cc233f12ab4f2d977c09d8abf5c9fb, State: Initialized, Role: FOLLOWER
I20260620 20:13:34.174896 26434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35cc233f12ab4f2d977c09d8abf5c9fb" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33721 } }
I20260620 20:13:34.175045 26434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:34.175127 26434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:34.175244 26434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:34.176345 26434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35cc233f12ab4f2d977c09d8abf5c9fb" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33721 } }
I20260620 20:13:34.176744 26434 leader_election.cc:302] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35cc233f12ab4f2d977c09d8abf5c9fb; no voters:
I20260620 20:13:34.177047 26434 leader_election.cc:288] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:34.177181 26439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:34.177415 26439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [term 1 LEADER]: Becoming Leader. State: Replica: 35cc233f12ab4f2d977c09d8abf5c9fb, State: Running, Role: LEADER
I20260620 20:13:34.177733 26439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35cc233f12ab4f2d977c09d8abf5c9fb" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33721 } }
I20260620 20:13:34.178422 26434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:34.179601 26441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35cc233f12ab4f2d977c09d8abf5c9fb. Latest consensus state: current_term: 1 leader_uuid: "35cc233f12ab4f2d977c09d8abf5c9fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35cc233f12ab4f2d977c09d8abf5c9fb" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33721 } } }
I20260620 20:13:34.179564 26440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35cc233f12ab4f2d977c09d8abf5c9fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35cc233f12ab4f2d977c09d8abf5c9fb" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33721 } } }
I20260620 20:13:34.179701 26440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:34.179701 26441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:34.180071 26443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:34.182006 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.166309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.24.20.190:33721)
0620 20:13:34.166749 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.166763 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.166797 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:13:34.167457 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.167474 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.167871 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.168231 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:34.169113 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.170136 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.171331 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.171838 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.171955 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.174991 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:34.174996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:34.175009 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:34.175060 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:34.176999 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.177591 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.177597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.177603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.177738 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.178051 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.178054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.178056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.178551 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:34.178724 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:34.180792 (+ 2068us) server_negotiation.cc:299] Negotiation successful
0620 20:13:34.181089 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":134,"threads_started":1}
I20260620 20:13:34.183058 26443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:34.192492 26443 catalog_manager.cc:1347] Generated new cluster ID: ce350584738e400382fcfbaedcd05dd9
I20260620 20:13:34.192575 26443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:34.206288 26443 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:34.207188 26443 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:34.214047 26443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb: Generated new TSK 0
I20260620 20:13:34.214828 26443 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:13:34.253199 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:33721
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:34.373106 26460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:34.373371 26460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:34.373418 26460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:34.377333 26460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:34.377380 26460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:34.377417 26460 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:13:34.377431 26460 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:13:34.377461 26460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:13:34.382786 26460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.20.190:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:34.383658 26460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:34.384545 26460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:34.393584 26468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:34.393615 26465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:34.393754 26460 server_base.cc:1034] running on GCE node
W20260620 20:13:34.393584 26466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:34.394261 26460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:34.394904 26460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:34.396082 26460 hybrid_clock.cc:648] HybridClock initialized: now 1781986414396066 us; error 28 us; skew 500 ppm
Jun 20 20:13:34 dist-test-slave-pflr krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986414, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:34.398769 26460 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:13:34.399739 26460 webserver.cc:460] Webserver started at http://127.24.20.129:46833/ using document root <none> and password file <none>
I20260620 20:13:34.400228 26460 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:34.400281 26460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:34.400432 26460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:34.402383 26460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "d90b85ee0494454dbfc41e43b6c7b551"
format_stamp: "Formatted at 2026-06-20 20:13:34 on dist-test-slave-pflr"
server_key: "e1b95f0e818cc8828cb3ffc1b1e2cab8"
server_key_iv: "be0059017e5e2dc50a76295475df3732"
server_key_version: "encryptionkey@0"
I20260620 20:13:34.402868 26460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "d90b85ee0494454dbfc41e43b6c7b551"
format_stamp: "Formatted at 2026-06-20 20:13:34 on dist-test-slave-pflr"
server_key: "e1b95f0e818cc8828cb3ffc1b1e2cab8"
server_key_iv: "be0059017e5e2dc50a76295475df3732"
server_key_version: "encryptionkey@0"
I20260620 20:13:34.406282 26460 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:13:34.408335 26475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:34.409533 26460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:34.409626 26460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "d90b85ee0494454dbfc41e43b6c7b551"
format_stamp: "Formatted at 2026-06-20 20:13:34 on dist-test-slave-pflr"
server_key: "e1b95f0e818cc8828cb3ffc1b1e2cab8"
server_key_iv: "be0059017e5e2dc50a76295475df3732"
server_key_version: "encryptionkey@0"
I20260620 20:13:34.409704 26460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:34.422771 26460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:34.425269 26460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:34.425398 26460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:34.426625 26460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:34.426676 26460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:34.426713 26460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:34.426779 26460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:13:34 dist-test-slave-pflr krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986414, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:34.437398 26460 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:41731
I20260620 20:13:34.437459 26590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:41731 every 8 connection(s)
I20260620 20:13:34.438397 26460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:34.439684 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.427298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:44249 (local address 127.24.20.190:33721)
0620 20:13:34.427450 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.427457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.428630 (+ 1173us) server_negotiation.cc:407] Connection header received
0620 20:13:34.429751 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.429754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.429811 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.429912 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:34.430929 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.431453 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.432701 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.433099 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.433217 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.435799 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:34.435802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:34.435805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:34.435832 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:34.437536 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.438092 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.438095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.438096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.438180 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.438558 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.438562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.438564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.438821 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:34.438912 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:34.439468 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 20:13:34.439562 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:13:34.440397 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.428061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33721 (local address 127.24.20.129:44249)
0620 20:13:34.428464 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:34.428510 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:34.429544 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:34.430067 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:34.430073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:34.430375 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:34.430781 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.430792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.431587 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.431591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.432535 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.432544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.433261 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.433265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.433474 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.434034 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:34.434060 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:34.435639 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:34.437673 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:34.437678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:34.437690 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:34.437987 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:34.438272 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:34.438275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:34.438277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:34.438441 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:34.438966 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:34.438974 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:34.439260 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 20:13:34.439409 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 20:13:34.439620 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":114,"threads_started":1}
I20260620 20:13:34.440871 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 26460
I20260620 20:13:34.440994 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:13:34.441329 24658 external_mini_cluster.cc:1428] Setting key cb937524aba6e2a8a699d5eb9bc8e092
I20260620 20:13:34.441798 26591 heartbeater.cc:347] Connected to a master server at 127.24.20.190:33721
I20260620 20:13:34.441942 26591 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:34.442503 26591 heartbeater.cc:511] Master 127.24.20.190:33721 requested a full tablet report, sending...
I20260620 20:13:34.445410 26387 ts_manager.cc:194] Registered new tserver with Master: d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129:41731)
I20260620 20:13:34.446863 26387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:44249
I20260620 20:13:34.451514 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.444260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:56447 (local address 127.24.20.190:33721)
0620 20:13:34.444397 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.444404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.444421 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:13:34.444491 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.444496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.444548 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.444665 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:34.445125 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.445845 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.446995 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.447231 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.447306 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.448450 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:34.448454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:34.448457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:34.448479 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:34.449952 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.450376 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.450381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.450383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.450461 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.450732 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.450737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.450738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.450900 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:34.450998 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:34.451254 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 20:13:34.451357 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:13:34.451520 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.443944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33721 (local address 127.24.20.129:56447)
0620 20:13:34.444317 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:34.444330 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:34.444411 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:34.444679 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:34.444682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:34.444879 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:34.444994 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.445001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.445997 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.446000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.446846 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.446856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.447342 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.447345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.447439 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.447982 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:34.448002 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:34.448310 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:34.450076 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:34.450080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:34.450082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:34.450279 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:34.450555 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:34.450558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:34.450560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:34.450645 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:34.450998 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:34.451002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:34.451091 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 20:13:34.451203 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 20:13:34.451334 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:13:34.471288 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:33721
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:34.582077 26600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:34.582347 26600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:34.582419 26600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:34.586452 26600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:34.586529 26600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:34.586601 26600 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:13:34.586648 26600 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:13:34.586714 26600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:13:34.591858 26600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.20.190:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:34.592896 26600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:34.593859 26600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:34.603152 26605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:34.603286 26600 server_base.cc:1034] running on GCE node
W20260620 20:13:34.603120 26608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:34.603120 26606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:34.603791 26600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:34.604429 26600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:34.605609 26600 hybrid_clock.cc:648] HybridClock initialized: now 1781986414605612 us; error 49 us; skew 500 ppm
Jun 20 20:13:34 dist-test-slave-pflr krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986414, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:34.608573 26600 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:13:34.609411 26600 webserver.cc:460] Webserver started at http://127.24.20.130:33139/ using document root <none> and password file <none>
I20260620 20:13:34.609912 26600 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:34.609968 26600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:34.610121 26600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:34.612013 26600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "f697d6be43894c6f9eb063522e44f51b"
format_stamp: "Formatted at 2026-06-20 20:13:34 on dist-test-slave-pflr"
server_key: "fe91398f4a89a483eeb5a39fb89d2483"
server_key_iv: "1c47045a60d42c2eb321c447153a009b"
server_key_version: "encryptionkey@0"
I20260620 20:13:34.612502 26600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "f697d6be43894c6f9eb063522e44f51b"
format_stamp: "Formatted at 2026-06-20 20:13:34 on dist-test-slave-pflr"
server_key: "fe91398f4a89a483eeb5a39fb89d2483"
server_key_iv: "1c47045a60d42c2eb321c447153a009b"
server_key_version: "encryptionkey@0"
I20260620 20:13:34.616107 26600 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:13:34.618484 26615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:34.619717 26600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:34.619839 26600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "f697d6be43894c6f9eb063522e44f51b"
format_stamp: "Formatted at 2026-06-20 20:13:34 on dist-test-slave-pflr"
server_key: "fe91398f4a89a483eeb5a39fb89d2483"
server_key_iv: "1c47045a60d42c2eb321c447153a009b"
server_key_version: "encryptionkey@0"
I20260620 20:13:34.619943 26600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:34.654572 26600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:34.658002 26600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:34.658214 26600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:34.659639 26600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:34.659686 26600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:34.659724 26600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:34.659778 26600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:13:34 dist-test-slave-pflr krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986414, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:34.670804 26600 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:38355
I20260620 20:13:34.670878 26730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:38355 every 8 connection(s)
I20260620 20:13:34.671914 26600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:34.674328 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.660640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:33173 (local address 127.24.20.190:33721)
0620 20:13:34.660776 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.660780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.661870 (+ 1090us) server_negotiation.cc:407] Connection header received
0620 20:13:34.663034 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.663039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.663102 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.663169 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:34.664928 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.665592 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.666743 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.667120 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.667265 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.670932 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:34.670936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:34.670939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:34.670965 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:34.672511 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.673098 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.673101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.673102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.673179 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.673462 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.673464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.673465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.673612 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:34.673681 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:34.674107 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 20:13:34.674209 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:13:34.675122 26624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.661194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33721 (local address 127.24.20.130:33173)
0620 20:13:34.661717 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:34.661759 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:34.662766 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:34.663641 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:34.663651 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:34.664038 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:34.664786 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.664808 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.665722 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.665726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.666615 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.666624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.667314 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.667322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.667564 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.668932 (+ 1368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:34.668956 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:34.670732 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:34.672670 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:34.672678 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:34.672693 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:34.672999 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:34.673262 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:34.673264 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:34.673270 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:34.673398 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:34.673694 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:34.673699 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:34.673898 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 20:13:34.674047 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 20:13:34.674274 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":131,"threads_started":1}
I20260620 20:13:34.676360 26731 heartbeater.cc:347] Connected to a master server at 127.24.20.190:33721
I20260620 20:13:34.676502 26731 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:34.676999 26731 heartbeater.cc:511] Master 127.24.20.190:33721 requested a full tablet report, sending...
I20260620 20:13:34.677556 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 26600
I20260620 20:13:34.677668 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:13:34.677963 26388 ts_manager.cc:194] Registered new tserver with Master: f697d6be43894c6f9eb063522e44f51b (127.24.20.130:38355)
I20260620 20:13:34.677999 24658 external_mini_cluster.cc:1428] Setting key d4bb13a560a38ea9c49f89b592b70ea9
I20260620 20:13:34.678660 26388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:33173
I20260620 20:13:34.686744 26737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.678617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33721 (local address 127.24.20.130:47081)
0620 20:13:34.679355 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:34.679369 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:34.679501 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:34.679868 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:34.679870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:34.680038 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:34.680125 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.680130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.681002 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.681005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.681625 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.681634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.682251 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.682255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.682333 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.682828 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:34.682848 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:34.683322 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:34.685339 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:34.685342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:34.685343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:34.685520 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:34.685822 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:34.685824 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:34.685826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:34.685893 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:34.686377 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:34.686379 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:34.686442 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 20:13:34.686514 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 20:13:34.686610 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":83,"threads_started":1}
I20260620 20:13:34.686888 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.678680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:47081 (local address 127.24.20.190:33721)
0620 20:13:34.678801 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.678805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.679443 (+ 638us) server_negotiation.cc:407] Connection header received
0620 20:13:34.679615 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.679620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.679671 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.679726 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:34.680237 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.680838 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.681738 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.682109 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.682236 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.683460 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:34.683462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:34.683464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:34.683484 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:34.685232 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.685620 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.685625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.685629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.685724 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.685992 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.685998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.686001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.686278 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:34.686384 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:34.686598 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 20:13:34.686711 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:13:34.706926 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:33721
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:34.814222 26740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:34.814452 26740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:34.814496 26740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:34.818363 26740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:34.818413 26740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:34.818449 26740 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:13:34.818463 26740 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:13:34.818497 26740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:13:34.823489 26740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.20.190:33721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:34.824473 26740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:34.825467 26740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:34.834338 26748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:34.834347 26746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:34.834357 26745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:34.834543 26740 server_base.cc:1034] running on GCE node
I20260620 20:13:34.835021 26740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:34.835613 26740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:34.836794 26740 hybrid_clock.cc:648] HybridClock initialized: now 1781986414836781 us; error 25 us; skew 500 ppm
Jun 20 20:13:34 dist-test-slave-pflr krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986414, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:34.839390 26740 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:13:34.840178 26740 webserver.cc:460] Webserver started at http://127.24.20.131:45303/ using document root <none> and password file <none>
I20260620 20:13:34.840781 26740 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:34.840870 26740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:34.841087 26740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:34.843096 26740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "9efceca70ade4461b426c1e4a1a72529"
format_stamp: "Formatted at 2026-06-20 20:13:34 on dist-test-slave-pflr"
server_key: "e69526af287cdc51de847542ef83e776"
server_key_iv: "d145e1a644d9f02b07e1fbd73990743e"
server_key_version: "encryptionkey@0"
I20260620 20:13:34.843621 26740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "9efceca70ade4461b426c1e4a1a72529"
format_stamp: "Formatted at 2026-06-20 20:13:34 on dist-test-slave-pflr"
server_key: "e69526af287cdc51de847542ef83e776"
server_key_iv: "d145e1a644d9f02b07e1fbd73990743e"
server_key_version: "encryptionkey@0"
I20260620 20:13:34.847129 26740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:13:34.849490 26755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:34.850606 26740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:34.850744 26740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "9efceca70ade4461b426c1e4a1a72529"
format_stamp: "Formatted at 2026-06-20 20:13:34 on dist-test-slave-pflr"
server_key: "e69526af287cdc51de847542ef83e776"
server_key_iv: "d145e1a644d9f02b07e1fbd73990743e"
server_key_version: "encryptionkey@0"
I20260620 20:13:34.850857 26740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:34.861961 26740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:34.864718 26740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:34.864907 26740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:34.866225 26740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:34.866297 26740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:34.866369 26740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:34.866420 26740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:13:34 dist-test-slave-pflr krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986414, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:34.876823 26740 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:46401
I20260620 20:13:34.876845 26870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:46401 every 8 connection(s)
I20260620 20:13:34.877701 26740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:13:34.880918 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.867056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:42563 (local address 127.24.20.190:33721)
0620 20:13:34.867192 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.867196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.868264 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 20:13:34.869349 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.869355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.869421 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.869485 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:34.871271 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.872172 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.873989 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.874202 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.874287 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.877172 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:34.877177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:34.877180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:34.877211 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:34.878892 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.879429 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.879432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.879434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.879516 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.879863 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.879866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.879868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.880045 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:34.880150 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:34.880636 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 20:13:34.880750 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:13:34.881712 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.867649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33721 (local address 127.24.20.131:42563)
0620 20:13:34.868119 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:34.868158 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:34.869101 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:34.869939 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:34.869947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:34.870395 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:34.871057 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.871078 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.872322 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.872329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.873876 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.873887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.874360 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.874364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.874570 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.875243 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:34.875265 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:34.876999 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:34.879003 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:34.879010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:34.879023 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:34.879323 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:34.879626 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:34.879630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:34.879632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:34.879764 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:34.880157 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:34.880165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:34.880380 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 20:13:34.880550 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 20:13:34.880855 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":128,"threads_started":1}
I20260620 20:13:34.882218 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 26740
I20260620 20:13:34.882354 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:13:34.882694 24658 external_mini_cluster.cc:1428] Setting key ccbf0c850256f67bf4ae5f68c5a9cd5c
I20260620 20:13:34.883111 26871 heartbeater.cc:347] Connected to a master server at 127.24.20.190:33721
I20260620 20:13:34.883265 26871 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:34.883713 26871 heartbeater.cc:511] Master 127.24.20.190:33721 requested a full tablet report, sending...
I20260620 20:13:34.884737 26388 ts_manager.cc:194] Registered new tserver with Master: 9efceca70ade4461b426c1e4a1a72529 (127.24.20.131:46401)
I20260620 20:13:34.885499 26388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:42563
I20260620 20:13:34.886929 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:13:34.892017 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.885009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:54047 (local address 127.24.20.190:33721)
0620 20:13:34.885111 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.885115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.885304 (+ 189us) server_negotiation.cc:407] Connection header received
0620 20:13:34.885389 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.885390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.885422 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.885481 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:34.885942 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.886500 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.887461 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.887690 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.887794 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.888928 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:34.888934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:34.888939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:34.888968 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:34.890486 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.890900 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.890904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.890906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.890985 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.891260 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.891265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.891266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.891425 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:34.891529 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:34.891768 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 20:13:34.891868 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:13:34.892026 26877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.884968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33721 (local address 127.24.20.131:54047)
0620 20:13:34.885219 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:34.885233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:34.885318 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:34.885520 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:34.885523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:34.885721 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:34.885831 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.885838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.886627 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.886632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.887340 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.887349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.887795 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.887798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.887889 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.888371 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:34.888384 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:34.888755 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:34.890612 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:34.890617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:34.890620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:34.890805 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:34.891080 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:34.891090 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:34.891092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:34.891173 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:34.891519 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:34.891522 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:34.891613 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 20:13:34.891717 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:13:34.891844 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":83,"threads_started":1}
Jun 20 20:13:34 dist-test-slave-pflr krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:13:34 dist-test-slave-pflr krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:34.913358 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.904731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.24.20.190:33721)
0620 20:13:34.904889 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.904894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.904911 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:13:34.904958 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.904961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.905015 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.905113 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:34.905775 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.906621 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.907503 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.907943 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.908068 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.910393 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:34.910396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:34.910398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:34.910420 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:34.911596 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.911953 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.911955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.911956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.912022 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:34.912244 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:34.912246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:34.912247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:34.912396 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:34.912488 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:34.913033 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 20:13:34.913153 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:13:34.916019 26388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:13:34.919011 26388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:34.932705 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.926949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:38355 (local address 127.0.0.1:44000)
0620 20:13:34.927539 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:34.927587 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:34.927709 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:34.928504 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:34.928510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:34.928533 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:34.928733 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.928741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.929955 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.929961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.931137 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.931146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.932149 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.932153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.932259 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.932294 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:13:34.932405 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:13:34.932513 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":67,"threads_started":1}
I20260620 20:13:34.933190 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.927061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.24.20.130:38355)
0620 20:13:34.927996 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.928009 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.928031 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:13:34.928125 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.928133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.928346 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.928508 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:34.928866 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.929853 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.931255 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.931834 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.931936 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.932089 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:34.932457 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 20:13:34.932569 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":98,"threads_started":1}
I20260620 20:13:34.933245 26888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.927603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:41731 (local address 127.0.0.1:38500)
0620 20:13:34.928793 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:34.928806 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:34.928892 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:34.929298 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:34.929301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:34.929318 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:34.929412 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.929417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.930677 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.930680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.931802 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.931813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.932813 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.932819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.932929 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.932952 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:13:34.933026 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:13:34.933107 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260620 20:13:34.933447 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.928378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.24.20.129:41731)
0620 20:13:34.928760 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.928775 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.928864 (+ 89us) server_negotiation.cc:407] Connection header received
0620 20:13:34.928993 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.928998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.929165 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.929310 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:34.929501 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.930550 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.931920 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.932490 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.932585 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.932745 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:34.933072 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 20:13:34.933170 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":69,"threads_started":1}
I20260620 20:13:34.933552 26892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.928031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.24.20.131:46401)
0620 20:13:34.928323 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.928332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.928349 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:13:34.928406 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.928410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.928598 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.928759 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:34.929183 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.930362 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.931824 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.932617 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.932713 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.932879 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:34.933316 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 20:13:34.933408 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
I20260620 20:13:34.933745 26891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.927016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:46401 (local address 127.0.0.1:46486)
0620 20:13:34.928001 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:34.928015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:34.928096 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:34.928902 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:34.928904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:34.928925 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:34.929053 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.929060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.930489 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.930493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.931695 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.931705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.932930 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.932933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.933034 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.933059 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:13:34.933138 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 20:13:34.933204 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":92,"threads_started":1}
I20260620 20:13:34.936218 26525 tablet_service.cc:1467] Processing CreateTablet for tablet 211a94ce477c4b5eb83e60185a5b34fe (DEFAULT_TABLE table=test-table [id=8ed3367fb2bc4396a8d75b07c2d90f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:34.936517 26665 tablet_service.cc:1467] Processing CreateTablet for tablet 211a94ce477c4b5eb83e60185a5b34fe (DEFAULT_TABLE table=test-table [id=8ed3367fb2bc4396a8d75b07c2d90f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:34.937247 26525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211a94ce477c4b5eb83e60185a5b34fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:34.937539 26665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211a94ce477c4b5eb83e60185a5b34fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:34.942615 26805 tablet_service.cc:1467] Processing CreateTablet for tablet 211a94ce477c4b5eb83e60185a5b34fe (DEFAULT_TABLE table=test-table [id=8ed3367fb2bc4396a8d75b07c2d90f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:34.943542 26805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211a94ce477c4b5eb83e60185a5b34fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:34.943935 26894 tablet_bootstrap.cc:492] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b: Bootstrap starting.
I20260620 20:13:34.944936 26895 tablet_bootstrap.cc:492] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551: Bootstrap starting.
I20260620 20:13:34.946194 26894 tablet_bootstrap.cc:654] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:34.946836 26895 tablet_bootstrap.cc:654] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:34.947055 26894 log.cc:824] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:34.947650 26895 log.cc:824] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:34.949075 26894 tablet_bootstrap.cc:492] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b: No bootstrap required, opened a new log
I20260620 20:13:34.949267 26894 ts_tablet_manager.cc:1397] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:13:34.949631 26895 tablet_bootstrap.cc:492] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551: No bootstrap required, opened a new log
I20260620 20:13:34.949822 26895 ts_tablet_manager.cc:1397] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:13:34.950510 26898 tablet_bootstrap.cc:492] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529: Bootstrap starting.
I20260620 20:13:34.952445 26898 tablet_bootstrap.cc:654] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:34.952795 26895 raft_consensus.cc:348] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } }
I20260620 20:13:34.953035 26895 raft_consensus.cc:374] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:34.953096 26895 raft_consensus.cc:729] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90b85ee0494454dbfc41e43b6c7b551, State: Initialized, Role: FOLLOWER
I20260620 20:13:34.953245 26898 log.cc:824] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:34.953130 26894 raft_consensus.cc:348] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } }
I20260620 20:13:34.953359 26894 raft_consensus.cc:374] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:34.953406 26894 raft_consensus.cc:729] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f697d6be43894c6f9eb063522e44f51b, State: Initialized, Role: FOLLOWER
I20260620 20:13:34.953595 26895 consensus_queue.cc:260] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } }
I20260620 20:13:34.953872 26894 consensus_queue.cc:260] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } }
I20260620 20:13:34.954068 26900 raft_consensus.cc:482] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:34.954249 26900 raft_consensus.cc:504] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } }
I20260620 20:13:34.954892 26894 ts_tablet_manager.cc:1428] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260620 20:13:34.955111 26898 tablet_bootstrap.cc:492] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529: No bootstrap required, opened a new log
I20260620 20:13:34.955235 26731 heartbeater.cc:503] Master 127.24.20.190:33721 was elected leader, sending a full tablet report...
I20260620 20:13:34.955278 26898 ts_tablet_manager.cc:1397] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:13:34.955564 26895 ts_tablet_manager.cc:1428] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 20:13:34.955891 26591 heartbeater.cc:503] Master 127.24.20.190:33721 was elected leader, sending a full tablet report...
I20260620 20:13:34.959074 26898 raft_consensus.cc:348] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } }
I20260620 20:13:34.959324 26898 raft_consensus.cc:374] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:34.959378 26898 raft_consensus.cc:729] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efceca70ade4461b426c1e4a1a72529, State: Initialized, Role: FOLLOWER
I20260620 20:13:34.959715 26900 leader_election.cc:288] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efceca70ade4461b426c1e4a1a72529 (127.24.20.131:46401), d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129:41731)
I20260620 20:13:34.959767 26898 consensus_queue.cc:260] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } }
I20260620 20:13:34.960695 26898 ts_tablet_manager.cc:1428] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:13:34.960976 26871 heartbeater.cc:503] Master 127.24.20.190:33721 was elected leader, sending a full tablet report...
I20260620 20:13:34.961470 26624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.956901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:46401 (local address 127.24.20.130:58575)
0620 20:13:34.957138 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:34.957165 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:34.957241 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:34.957815 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:34.957818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:34.957840 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:34.957963 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.957970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.958886 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.958889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.960077 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.960097 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.961057 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.961061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.961184 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.961206 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:13:34.961281 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 20:13:34.961352 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 20:13:34.961906 26892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.957385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:58575 (local address 127.24.20.131:46401)
0620 20:13:34.957507 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.957512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.957525 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:13:34.957569 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.957574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.957618 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.957689 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:34.958093 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.958776 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.960226 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.960823 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.960946 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.960997 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:34.961712 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 20:13:34.961787 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":56}
I20260620 20:13:34.962383 26825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211a94ce477c4b5eb83e60185a5b34fe" candidate_uuid: "f697d6be43894c6f9eb063522e44f51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efceca70ade4461b426c1e4a1a72529" is_pre_election: true
I20260620 20:13:34.962646 26825 raft_consensus.cc:2454] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f697d6be43894c6f9eb063522e44f51b in term 0.
I20260620 20:13:34.963112 26617 leader_election.cc:302] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9efceca70ade4461b426c1e4a1a72529, f697d6be43894c6f9eb063522e44f51b; no voters:
I20260620 20:13:34.963536 26902 raft_consensus.cc:2781] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:34.963671 26902 raft_consensus.cc:482] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:34.963737 26902 raft_consensus.cc:3037] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:34.964443 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.958042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:45799 (local address 127.24.20.129:41731)
0620 20:13:34.958143 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.958148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.960131 (+ 1983us) server_negotiation.cc:407] Connection header received
0620 20:13:34.960295 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.960298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.960347 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.960395 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:34.961175 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.961883 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.963313 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.963898 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.963970 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.964003 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:34.964283 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 20:13:34.964346 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:13:34.964460 26903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.957738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:41731 (local address 127.24.20.130:45799)
0620 20:13:34.960049 (+ 2311us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:34.960063 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:34.960185 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:34.960545 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:34.960548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:34.960564 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:34.961039 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.961048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.962021 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.962025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.963162 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.963171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.964094 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.964096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.964163 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.964176 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 20:13:34.964257 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 20:13:34.964320 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2252,"thread_start_us":90,"threads_started":1}
I20260620 20:13:34.964969 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211a94ce477c4b5eb83e60185a5b34fe" candidate_uuid: "f697d6be43894c6f9eb063522e44f51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90b85ee0494454dbfc41e43b6c7b551" is_pre_election: true
I20260620 20:13:34.965152 26902 raft_consensus.cc:504] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } }
I20260620 20:13:34.965236 26545 raft_consensus.cc:2454] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f697d6be43894c6f9eb063522e44f51b in term 0.
I20260620 20:13:34.965492 26902 leader_election.cc:288] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [CANDIDATE]: Term 1 election: Requested vote from peers 9efceca70ade4461b426c1e4a1a72529 (127.24.20.131:46401), d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129:41731)
I20260620 20:13:34.965818 26825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211a94ce477c4b5eb83e60185a5b34fe" candidate_uuid: "f697d6be43894c6f9eb063522e44f51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efceca70ade4461b426c1e4a1a72529"
I20260620 20:13:34.965929 26825 raft_consensus.cc:3037] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:34.965925 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211a94ce477c4b5eb83e60185a5b34fe" candidate_uuid: "f697d6be43894c6f9eb063522e44f51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90b85ee0494454dbfc41e43b6c7b551"
I20260620 20:13:34.966058 26545 raft_consensus.cc:3037] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:34.966887 26825 raft_consensus.cc:2454] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f697d6be43894c6f9eb063522e44f51b in term 1.
I20260620 20:13:34.966960 26545 raft_consensus.cc:2454] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f697d6be43894c6f9eb063522e44f51b in term 1.
I20260620 20:13:34.967159 26617 leader_election.cc:302] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9efceca70ade4461b426c1e4a1a72529, f697d6be43894c6f9eb063522e44f51b; no voters:
I20260620 20:13:34.967314 26902 raft_consensus.cc:2781] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:34.967430 26902 raft_consensus.cc:686] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [term 1 LEADER]: Becoming Leader. State: Replica: f697d6be43894c6f9eb063522e44f51b, State: Running, Role: LEADER
I20260620 20:13:34.967707 26902 consensus_queue.cc:237] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } }
I20260620 20:13:34.970451 26388 catalog_manager.cc:5495] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b reported cstate change: term changed from 0 to 1, leader changed from <none> to f697d6be43894c6f9eb063522e44f51b (127.24.20.130). New cstate: current_term: 1 leader_uuid: "f697d6be43894c6f9eb063522e44f51b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } health_report { overall_health: UNKNOWN } } }
Jun 20 20:13:34 dist-test-slave-pflr krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986414, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:35.001785 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.994158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33721 (local address 127.0.0.1:39200)
0620 20:13:34.994437 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:34.994455 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:34.994560 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:34.994784 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:34.994787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:34.995147 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:34.995235 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.995240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.995912 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.995914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.996449 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:34.996455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.996878 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.996880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:34.996938 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.997439 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:34.997450 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:34.998775 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:35.000392 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:35.000399 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:35.000404 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:35.000689 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:35.000920 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:35.000922 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:35.000923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:35.000995 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:35.001314 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:35.001319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:35.001408 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 20:13:35.001495 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 20:13:35.001609 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":122,"threads_started":1}
I20260620 20:13:35.001793 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:34.994207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.24.20.190:33721)
0620 20:13:34.994318 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:34.994322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:34.994530 (+ 208us) server_negotiation.cc:407] Connection header received
0620 20:13:34.994639 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:34.994641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:34.994677 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:34.994753 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:34.995328 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.995826 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.996543 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:34.996784 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:34.996858 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:34.998891 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:34.998894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:34.998896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:34.998916 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:35.000298 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:35.000771 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:35.000773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:35.000775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:35.000843 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:35.001068 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:35.001070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:35.001072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:35.001229 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:35.001300 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:35.001547 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 20:13:35.001671 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":44}
I20260620 20:13:35.005600 26388 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:39200:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:13:35.006300 26388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:35.010782 26525 tablet_service.cc:1467] Processing CreateTablet for tablet a08a97e2e8b742c5a569438bcfe053fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fee752aaf88a48ca9847a8188ecb14bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:13:35.010895 26665 tablet_service.cc:1467] Processing CreateTablet for tablet a08a97e2e8b742c5a569438bcfe053fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fee752aaf88a48ca9847a8188ecb14bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:13:35.011535 26525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08a97e2e8b742c5a569438bcfe053fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:35.011559 26665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08a97e2e8b742c5a569438bcfe053fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:35.014232 26894 tablet_bootstrap.cc:492] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b: Bootstrap starting.
I20260620 20:13:35.014247 26895 tablet_bootstrap.cc:492] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551: Bootstrap starting.
I20260620 20:13:35.015007 26895 tablet_bootstrap.cc:654] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:35.015188 26894 tablet_bootstrap.cc:654] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:35.014855 26805 tablet_service.cc:1467] Processing CreateTablet for tablet a08a97e2e8b742c5a569438bcfe053fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fee752aaf88a48ca9847a8188ecb14bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:13:35.015475 26805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a08a97e2e8b742c5a569438bcfe053fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:35.016461 26894 tablet_bootstrap.cc:492] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b: No bootstrap required, opened a new log
I20260620 20:13:35.016562 26894 ts_tablet_manager.cc:1397] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 20:13:35.017072 26894 raft_consensus.cc:348] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.017192 26894 raft_consensus.cc:374] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:35.017230 26894 raft_consensus.cc:729] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f697d6be43894c6f9eb063522e44f51b, State: Initialized, Role: FOLLOWER
I20260620 20:13:35.017351 26894 consensus_queue.cc:260] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.017546 26895 tablet_bootstrap.cc:492] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551: No bootstrap required, opened a new log
I20260620 20:13:35.017625 26895 ts_tablet_manager.cc:1397] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 20:13:35.017583 26902 tablet_replica.cc:439] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } }
I20260620 20:13:35.017704 26902 tablet_replica.cc:442] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:13:35.018069 26894 ts_tablet_manager.cc:1428] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:35.018056 26895 raft_consensus.cc:348] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.018173 26895 raft_consensus.cc:374] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:35.018208 26895 raft_consensus.cc:729] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90b85ee0494454dbfc41e43b6c7b551, State: Initialized, Role: FOLLOWER
I20260620 20:13:35.018296 26898 tablet_bootstrap.cc:492] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529: Bootstrap starting.
I20260620 20:13:35.018313 26895 consensus_queue.cc:260] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.018568 26895 ts_tablet_manager.cc:1428] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:35.018811 26901 tablet_replica.cc:439] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } }
I20260620 20:13:35.018940 26901 tablet_replica.cc:442] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:13:35.019263 26898 tablet_bootstrap.cc:654] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:35.020713 26898 tablet_bootstrap.cc:492] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529: No bootstrap required, opened a new log
I20260620 20:13:35.020795 26898 ts_tablet_manager.cc:1397] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:13:35.021266 26898 raft_consensus.cc:348] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.021371 26898 raft_consensus.cc:374] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:35.021413 26898 raft_consensus.cc:729] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efceca70ade4461b426c1e4a1a72529, State: Initialized, Role: FOLLOWER
I20260620 20:13:35.021533 26898 consensus_queue.cc:260] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.021812 26898 ts_tablet_manager.cc:1428] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 20:13:35.021783 26904 tablet_replica.cc:439] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } }
I20260620 20:13:35.021910 26904 tablet_replica.cc:442] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 20:13:35.128826 26872 tablet.cc:2369] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:13:35.168805 26904 raft_consensus.cc:482] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:35.168947 26904 raft_consensus.cc:504] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.170027 26904 leader_election.cc:288] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129:41731), f697d6be43894c6f9eb063522e44f51b (127.24.20.130:38355)
W20260620 20:13:35.173050 26732 tablet.cc:2369] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:13:35.173743 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.170394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:34951 (local address 127.24.20.129:41731)
0620 20:13:35.170538 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:35.170544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:35.170564 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:13:35.170755 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:35.170762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:35.170817 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:35.170899 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:35.171183 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.171680 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.172696 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.173218 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.173335 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.173385 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:35.173569 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 20:13:35.173644 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:13:35.173761 26764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.170340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:41731 (local address 127.24.20.131:34951)
0620 20:13:35.170497 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:35.170513 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:35.170620 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:35.170915 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:35.170918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:35.170934 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:35.171055 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.171060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.171788 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.171794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.172553 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.172561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.173331 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.173335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.173438 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.173462 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:13:35.173538 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 20:13:35.173604 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:13:35.174294 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08a97e2e8b742c5a569438bcfe053fc" candidate_uuid: "9efceca70ade4461b426c1e4a1a72529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90b85ee0494454dbfc41e43b6c7b551" is_pre_election: true
I20260620 20:13:35.174444 26915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.170621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:38355 (local address 127.24.20.131:35633)
0620 20:13:35.170964 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:35.170976 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:35.171093 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:35.171347 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:35.171349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:35.171363 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:35.171496 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.171501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.172212 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.172216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.173077 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.173086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.174057 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.174061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.174157 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.174183 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 20:13:35.174251 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 20:13:35.174316 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":124,"threads_started":1}
I20260620 20:13:35.174439 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.170771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:35633 (local address 127.24.20.130:38355)
0620 20:13:35.170899 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:35.170903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:35.171061 (+ 158us) server_negotiation.cc:407] Connection header received
0620 20:13:35.171201 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:35.171204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:35.171247 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:35.171336 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:35.171607 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.172097 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.173239 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.173817 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.173909 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.173958 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:35.174276 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 20:13:35.174342 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:13:35.174448 26545 raft_consensus.cc:2454] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efceca70ade4461b426c1e4a1a72529 in term 0.
I20260620 20:13:35.174853 26759 leader_election.cc:302] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9efceca70ade4461b426c1e4a1a72529, d90b85ee0494454dbfc41e43b6c7b551; no voters:
I20260620 20:13:35.175122 26904 raft_consensus.cc:2781] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:35.175060 26685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08a97e2e8b742c5a569438bcfe053fc" candidate_uuid: "9efceca70ade4461b426c1e4a1a72529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f697d6be43894c6f9eb063522e44f51b" is_pre_election: true
I20260620 20:13:35.175210 26904 raft_consensus.cc:482] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:35.175249 26904 raft_consensus.cc:3037] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:35.175310 26685 raft_consensus.cc:2454] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efceca70ade4461b426c1e4a1a72529 in term 0.
I20260620 20:13:35.176476 26904 raft_consensus.cc:504] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.176815 26904 leader_election.cc:288] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [CANDIDATE]: Term 1 election: Requested vote from peers d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129:41731), f697d6be43894c6f9eb063522e44f51b (127.24.20.130:38355)
I20260620 20:13:35.177103 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08a97e2e8b742c5a569438bcfe053fc" candidate_uuid: "9efceca70ade4461b426c1e4a1a72529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90b85ee0494454dbfc41e43b6c7b551"
I20260620 20:13:35.177145 26685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08a97e2e8b742c5a569438bcfe053fc" candidate_uuid: "9efceca70ade4461b426c1e4a1a72529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f697d6be43894c6f9eb063522e44f51b"
I20260620 20:13:35.177240 26685 raft_consensus.cc:3037] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:35.177260 26545 raft_consensus.cc:3037] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:35.178361 26685 raft_consensus.cc:2454] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efceca70ade4461b426c1e4a1a72529 in term 1.
I20260620 20:13:35.178362 26545 raft_consensus.cc:2454] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efceca70ade4461b426c1e4a1a72529 in term 1.
I20260620 20:13:35.178634 26759 leader_election.cc:302] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9efceca70ade4461b426c1e4a1a72529, d90b85ee0494454dbfc41e43b6c7b551; no voters:
I20260620 20:13:35.178846 26904 raft_consensus.cc:2781] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:35.179059 26904 raft_consensus.cc:686] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [term 1 LEADER]: Becoming Leader. State: Replica: 9efceca70ade4461b426c1e4a1a72529, State: Running, Role: LEADER
I20260620 20:13:35.179397 26904 consensus_queue.cc:237] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.181665 26916 tablet_replica.cc:439] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529: TxnStatusTablet state changed. Reason: New leader 9efceca70ade4461b426c1e4a1a72529. Latest consensus state: current_term: 1 leader_uuid: "9efceca70ade4461b426c1e4a1a72529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } }
I20260620 20:13:35.181775 26916 tablet_replica.cc:442] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:13:35.182044 26918 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:13:35.182297 26918 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:13:35.182734 26388 catalog_manager.cc:5495] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9efceca70ade4461b426c1e4a1a72529 (127.24.20.131). New cstate: current_term: 1 leader_uuid: "9efceca70ade4461b426c1e4a1a72529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } health_report { overall_health: HEALTHY } } }
W20260620 20:13:35.189646 26592 tablet.cc:2369] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:13:35.205806 26892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.202202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46494 (local address 127.24.20.131:46401)
0620 20:13:35.202325 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:35.202330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:35.202342 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:13:35.202528 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:35.202532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:35.202591 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:35.202699 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:35.202930 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.203540 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.204366 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.204588 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.204746 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.205028 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:35.205411 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:35.205532 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:35.205607 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 20:13:35.205658 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:13:35.205803 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.202102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:46401 (local address 127.0.0.1:46494)
0620 20:13:35.202247 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:35.202264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:35.202373 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:35.202710 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:35.202713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:35.202722 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:13:35.202828 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.202833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.203650 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.203653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.204250 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.204257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.204743 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.204746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.204847 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.204887 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:13:35.205506 (+ 619us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:13:35.205521 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 20:13:35.205588 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 20:13:35.205681 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 20:13:35.210812 26685 raft_consensus.cc:1261] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Refusing update from remote peer 9efceca70ade4461b426c1e4a1a72529: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:35.210814 26545 raft_consensus.cc:1261] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Refusing update from remote peer 9efceca70ade4461b426c1e4a1a72529: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:35.211421 26904 consensus_queue.cc:1035] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:35.211577 26916 consensus_queue.cc:1035] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:35.214746 26902 tablet_replica.cc:439] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b: TxnStatusTablet state changed. Reason: New leader 9efceca70ade4461b426c1e4a1a72529. Latest consensus state: current_term: 1 leader_uuid: "9efceca70ade4461b426c1e4a1a72529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } }
I20260620 20:13:35.214861 26902 tablet_replica.cc:442] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:13:35.216810 26901 tablet_replica.cc:439] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551: TxnStatusTablet state changed. Reason: New leader 9efceca70ade4461b426c1e4a1a72529. Latest consensus state: current_term: 1 leader_uuid: "9efceca70ade4461b426c1e4a1a72529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } }
I20260620 20:13:35.216926 26901 tablet_replica.cc:442] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:13:35.217445 26921 mvcc.cc:205] Tried to move back new op lower bound from 7299016356696989696 to 7299016356577501184. Current Snapshot: MvccSnapshot[applied={T|T < 7299016356696989696}]
I20260620 20:13:35.217551 26904 tablet_replica.cc:439] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9efceca70ade4461b426c1e4a1a72529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } }
I20260620 20:13:35.217670 26904 tablet_replica.cc:442] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:13:35.218150 26902 tablet_replica.cc:439] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9efceca70ade4461b426c1e4a1a72529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } }
I20260620 20:13:35.218252 26902 tablet_replica.cc:442] T a08a97e2e8b742c5a569438bcfe053fc P f697d6be43894c6f9eb063522e44f51b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:13:35.220108 26916 tablet_replica.cc:439] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9efceca70ade4461b426c1e4a1a72529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } }
I20260620 20:13:35.220216 26916 tablet_replica.cc:442] T a08a97e2e8b742c5a569438bcfe053fc P 9efceca70ade4461b426c1e4a1a72529: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:13:35.221221 26901 tablet_replica.cc:439] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9efceca70ade4461b426c1e4a1a72529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } } }
I20260620 20:13:35.221323 26901 tablet_replica.cc:442] T a08a97e2e8b742c5a569438bcfe053fc P d90b85ee0494454dbfc41e43b6c7b551: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:13:35.224834 26923 mvcc.cc:205] Tried to move back new op lower bound from 7299016356696989696 to 7299016356577501184. Current Snapshot: MvccSnapshot[applied={T|T < 7299016356696989696}]
I20260620 20:13:35.229940 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.226230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.24.20.130:38355)
0620 20:13:35.226332 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:35.226337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:35.226390 (+ 53us) server_negotiation.cc:407] Connection header received
0620 20:13:35.226536 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:35.226538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:35.226582 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:35.226680 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:35.226922 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.227429 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.228676 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.228896 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.228973 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.229242 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:35.229586 (+ 344us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:35.229669 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:35.229765 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 20:13:35.229806 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 20:13:35.248075 26892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.244671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:59235 (local address 127.24.20.131:46401)
0620 20:13:35.244800 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:35.244804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:35.244967 (+ 163us) server_negotiation.cc:407] Connection header received
0620 20:13:35.245073 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:35.245076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:35.245117 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:35.245180 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:35.245447 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.245947 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.247126 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.247327 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.247395 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.247722 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:35.247781 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:35.247829 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:35.247939 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 20:13:35.247978 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:13:35.248147 26933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.244555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:46401 (local address 127.24.20.130:59235)
0620 20:13:35.244883 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:35.244898 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:35.244996 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:35.245211 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:35.245215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:35.245223 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:13:35.245338 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.245344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.246042 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.246045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.247013 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.247023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.247429 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.247432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.247521 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.247567 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:13:35.247854 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:13:35.247860 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:13:35.247925 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 20:13:35.247996 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":153,"threads_started":1}
I20260620 20:13:35.254869 26545 raft_consensus.cc:1261] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Refusing update from remote peer f697d6be43894c6f9eb063522e44f51b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:35.255194 26825 raft_consensus.cc:1261] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Refusing update from remote peer f697d6be43894c6f9eb063522e44f51b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:35.255362 26900 consensus_queue.cc:1035] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:35.255633 26902 consensus_queue.cc:1035] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:35.261426 26922 mvcc.cc:205] Tried to move back new op lower bound from 7299016356880060416 to 7299016355710328832. Current Snapshot: MvccSnapshot[applied={T|T < 7299016356880060416}]
I20260620 20:13:35.277787 26877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.274213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:38355 (local address 127.24.20.131:40661)
0620 20:13:35.274354 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:35.274368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:35.274451 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:35.274668 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:35.274670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:35.274674 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:13:35.274766 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.274770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.275779 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.275783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.276462 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.276491 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.277099 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.277101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.277225 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.277267 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:13:35.277573 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:13:35.277578 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:13:35.277629 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 20:13:35.277689 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 20:13:35.277808 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.274276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:40661 (local address 127.24.20.130:38355)
0620 20:13:35.274379 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:35.274384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:35.274437 (+ 53us) server_negotiation.cc:407] Connection header received
0620 20:13:35.274536 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:35.274539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:35.274585 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:35.274666 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:35.274865 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.275674 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.276606 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.277000 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.277123 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.277382 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:35.277489 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:35.277558 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:35.277642 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 20:13:35.277691 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:13:35.299371 26387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:35.299556 26387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:35.302196 26387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb: Sending DeleteTablet for 3 replicas of tablet 211a94ce477c4b5eb83e60185a5b34fe
I20260620 20:13:35.302958 26665 tablet_service.cc:1514] Processing DeleteTablet for tablet 211a94ce477c4b5eb83e60185a5b34fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:35 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:44000
I20260620 20:13:35.303097 26525 tablet_service.cc:1514] Processing DeleteTablet for tablet 211a94ce477c4b5eb83e60185a5b34fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:35 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:38500
I20260620 20:13:35.303737 26805 tablet_service.cc:1514] Processing DeleteTablet for tablet 211a94ce477c4b5eb83e60185a5b34fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:35 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:46486
I20260620 20:13:35.304198 26942 tablet_replica.cc:330] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551: stopping tablet replica
I20260620 20:13:35.304301 26387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 20:13:35.304494 26942 raft_consensus.cc:2229] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 20:13:35.304683 26387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:35.304729 26942 raft_consensus.cc:2258] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:35.305846 26943 tablet_replica.cc:330] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529: stopping tablet replica
I20260620 20:13:35.306177 26943 raft_consensus.cc:2229] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:35.306380 26943 raft_consensus.cc:2258] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:35.307435 26942 ts_tablet_manager.cc:1905] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:35.308632 26943 ts_tablet_manager.cc:1905] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:35.308748 26941 tablet_replica.cc:330] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b: stopping tablet replica
I20260620 20:13:35.309003 26941 raft_consensus.cc:2229] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:13:35.309345 26941 raft_consensus.cc:2258] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:35.310262 26525 tablet_service.cc:1467] Processing CreateTablet for tablet 42eadd09bf2846699de998531692198a (DEFAULT_TABLE table=test-table [id=040013c5e0714a65934165c071b118d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:35.310463 26942 ts_tablet_manager.cc:1918] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:13:35.310518 26525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42eadd09bf2846699de998531692198a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:35.310549 26942 log.cc:1196] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/wal/wals/211a94ce477c4b5eb83e60185a5b34fe
I20260620 20:13:35.310575 26941 ts_tablet_manager.cc:1905] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:35.310876 26942 ts_tablet_manager.cc:1939] T 211a94ce477c4b5eb83e60185a5b34fe P d90b85ee0494454dbfc41e43b6c7b551: Deleting consensus metadata
I20260620 20:13:35.312005 26375 catalog_manager.cc:4841] TS d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129:41731): tablet 211a94ce477c4b5eb83e60185a5b34fe (table test-table [id=8ed3367fb2bc4396a8d75b07c2d90f22]) successfully deleted
I20260620 20:13:35.312810 26665 tablet_service.cc:1467] Processing CreateTablet for tablet 42eadd09bf2846699de998531692198a (DEFAULT_TABLE table=test-table [id=040013c5e0714a65934165c071b118d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:35.313069 26665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42eadd09bf2846699de998531692198a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:35.313235 26943 ts_tablet_manager.cc:1918] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:13:35.313366 26943 log.cc:1196] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/211a94ce477c4b5eb83e60185a5b34fe
I20260620 20:13:35.313692 26943 ts_tablet_manager.cc:1939] T 211a94ce477c4b5eb83e60185a5b34fe P 9efceca70ade4461b426c1e4a1a72529: Deleting consensus metadata
I20260620 20:13:35.314803 26373 catalog_manager.cc:4841] TS 9efceca70ade4461b426c1e4a1a72529 (127.24.20.131:46401): tablet 211a94ce477c4b5eb83e60185a5b34fe (table test-table [id=8ed3367fb2bc4396a8d75b07c2d90f22]) successfully deleted
I20260620 20:13:35.315091 26894 tablet_bootstrap.cc:492] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b: Bootstrap starting.
I20260620 20:13:35.315260 26805 tablet_service.cc:1467] Processing CreateTablet for tablet 42eadd09bf2846699de998531692198a (DEFAULT_TABLE table=test-table [id=040013c5e0714a65934165c071b118d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:35.315510 26805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42eadd09bf2846699de998531692198a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:35.316118 26894 tablet_bootstrap.cc:654] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:35.317333 26894 tablet_bootstrap.cc:492] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b: No bootstrap required, opened a new log
I20260620 20:13:35.317409 26898 tablet_bootstrap.cc:492] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529: Bootstrap starting.
I20260620 20:13:35.317412 26894 ts_tablet_manager.cc:1397] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:35.317490 26895 tablet_bootstrap.cc:492] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551: Bootstrap starting.
I20260620 20:13:35.317827 26894 raft_consensus.cc:348] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.317929 26894 raft_consensus.cc:374] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:35.317958 26894 raft_consensus.cc:729] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f697d6be43894c6f9eb063522e44f51b, State: Initialized, Role: FOLLOWER
I20260620 20:13:35.318085 26894 consensus_queue.cc:260] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.318320 26898 tablet_bootstrap.cc:654] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:35.318325 26894 ts_tablet_manager.cc:1428] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:35.318338 26895 tablet_bootstrap.cc:654] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:35.319543 26895 tablet_bootstrap.cc:492] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551: No bootstrap required, opened a new log
I20260620 20:13:35.319620 26895 ts_tablet_manager.cc:1397] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:35.320055 26895 raft_consensus.cc:348] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.320195 26895 raft_consensus.cc:374] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:35.320235 26895 raft_consensus.cc:729] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90b85ee0494454dbfc41e43b6c7b551, State: Initialized, Role: FOLLOWER
I20260620 20:13:35.320379 26895 consensus_queue.cc:260] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.320627 26895 ts_tablet_manager.cc:1428] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:35.322446 26898 tablet_bootstrap.cc:492] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529: No bootstrap required, opened a new log
I20260620 20:13:35.322522 26898 ts_tablet_manager.cc:1397] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 20:13:35.322926 26898 raft_consensus.cc:348] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.323030 26898 raft_consensus.cc:374] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:35.323060 26898 raft_consensus.cc:729] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efceca70ade4461b426c1e4a1a72529, State: Initialized, Role: FOLLOWER
I20260620 20:13:35.323168 26898 consensus_queue.cc:260] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.323434 26898 ts_tablet_manager.cc:1428] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:35.326018 26941 ts_tablet_manager.cc:1918] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:13:35.326155 26941 log.cc:1196] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/211a94ce477c4b5eb83e60185a5b34fe
I20260620 20:13:35.326501 26941 ts_tablet_manager.cc:1939] T 211a94ce477c4b5eb83e60185a5b34fe P f697d6be43894c6f9eb063522e44f51b: Deleting consensus metadata
I20260620 20:13:35.327725 26375 catalog_manager.cc:4841] TS f697d6be43894c6f9eb063522e44f51b (127.24.20.130:38355): tablet 211a94ce477c4b5eb83e60185a5b34fe (table test-table [id=8ed3367fb2bc4396a8d75b07c2d90f22]) successfully deleted
I20260620 20:13:35.332827 26901 raft_consensus.cc:482] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:35.332937 26901 raft_consensus.cc:504] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.334108 26901 leader_election.cc:288] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f697d6be43894c6f9eb063522e44f51b (127.24.20.130:38355), 9efceca70ade4461b426c1e4a1a72529 (127.24.20.131:46401)
I20260620 20:13:35.339283 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.334533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:34463 (local address 127.24.20.130:38355)
0620 20:13:35.334745 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:35.334750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:35.334817 (+ 67us) server_negotiation.cc:407] Connection header received
0620 20:13:35.334995 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:35.334998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:35.335038 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:35.335081 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:35.335465 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.336239 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.338004 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.338585 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.338682 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.338732 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:35.339094 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 20:13:35.339162 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260620 20:13:35.339838 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.334259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:38355 (local address 127.24.20.129:34463)
0620 20:13:35.334734 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:35.334749 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:35.334883 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:35.335216 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:35.335219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:35.335234 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:35.335352 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.335357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.336416 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.336419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.337867 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.337878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.338845 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.338849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.338964 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.338986 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:13:35.339596 (+ 610us) client_negotiation.cc:240] Negotiation successful
0620 20:13:35.339683 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":13,"spinlock_wait_cycles":7936,"thread_start_us":70,"threads_started":1}
I20260620 20:13:35.340327 26892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.334848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:42753 (local address 127.24.20.131:46401)
0620 20:13:35.334958 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:35.334962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:35.335195 (+ 233us) server_negotiation.cc:407] Connection header received
0620 20:13:35.335406 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:35.335409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:35.335451 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:35.335500 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:35.335898 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.336680 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.339076 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.339638 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.339732 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.339779 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:35.340141 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 20:13:35.340216 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:13:35.340361 26684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eadd09bf2846699de998531692198a" candidate_uuid: "d90b85ee0494454dbfc41e43b6c7b551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f697d6be43894c6f9eb063522e44f51b" is_pre_election: true
I20260620 20:13:35.340490 26684 raft_consensus.cc:2454] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d90b85ee0494454dbfc41e43b6c7b551 in term 0.
I20260620 20:13:35.340737 26948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.334395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:46401 (local address 127.24.20.129:42753)
0620 20:13:35.335102 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:35.335116 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:35.335284 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:35.335634 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:35.335637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:35.335654 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:35.335769 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.335776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.336853 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.336856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.338912 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.338924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.339878 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.339881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.339978 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.340000 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:13:35.340503 (+ 503us) client_negotiation.cc:240] Negotiation successful
0620 20:13:35.340573 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"spinlock_wait_cycles":23936,"thread_start_us":92,"threads_started":1}
I20260620 20:13:35.341027 26479 leader_election.cc:302] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d90b85ee0494454dbfc41e43b6c7b551, f697d6be43894c6f9eb063522e44f51b; no voters:
I20260620 20:13:35.341282 26901 raft_consensus.cc:2781] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:35.341344 26901 raft_consensus.cc:482] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:35.341320 26825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eadd09bf2846699de998531692198a" candidate_uuid: "d90b85ee0494454dbfc41e43b6c7b551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efceca70ade4461b426c1e4a1a72529" is_pre_election: true
I20260620 20:13:35.341383 26901 raft_consensus.cc:3037] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:35.341436 26825 raft_consensus.cc:2454] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d90b85ee0494454dbfc41e43b6c7b551 in term 0.
I20260620 20:13:35.342667 26901 raft_consensus.cc:504] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.343024 26901 leader_election.cc:288] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [CANDIDATE]: Term 1 election: Requested vote from peers f697d6be43894c6f9eb063522e44f51b (127.24.20.130:38355), 9efceca70ade4461b426c1e4a1a72529 (127.24.20.131:46401)
I20260620 20:13:35.343569 26825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eadd09bf2846699de998531692198a" candidate_uuid: "d90b85ee0494454dbfc41e43b6c7b551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efceca70ade4461b426c1e4a1a72529"
I20260620 20:13:35.343686 26825 raft_consensus.cc:3037] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:35.344755 26825 raft_consensus.cc:2454] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d90b85ee0494454dbfc41e43b6c7b551 in term 1.
I20260620 20:13:35.345108 26684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42eadd09bf2846699de998531692198a" candidate_uuid: "d90b85ee0494454dbfc41e43b6c7b551" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f697d6be43894c6f9eb063522e44f51b"
I20260620 20:13:35.345214 26684 raft_consensus.cc:3037] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:35.345259 26477 leader_election.cc:302] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9efceca70ade4461b426c1e4a1a72529, d90b85ee0494454dbfc41e43b6c7b551; no voters:
I20260620 20:13:35.345450 26901 raft_consensus.cc:2781] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:35.345649 26901 raft_consensus.cc:686] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [term 1 LEADER]: Becoming Leader. State: Replica: d90b85ee0494454dbfc41e43b6c7b551, State: Running, Role: LEADER
I20260620 20:13:35.345890 26901 consensus_queue.cc:237] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:35.346289 26684 raft_consensus.cc:2454] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d90b85ee0494454dbfc41e43b6c7b551 in term 1.
I20260620 20:13:35.348162 26387 catalog_manager.cc:5495] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 reported cstate change: term changed from 0 to 1, leader changed from <none> to d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129). New cstate: current_term: 1 leader_uuid: "d90b85ee0494454dbfc41e43b6c7b551" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } health_report { overall_health: UNKNOWN } } }
I20260620 20:13:35.367327 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.362739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.24.20.129:41731)
0620 20:13:35.362947 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:35.362952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:35.362964 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:13:35.363046 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:35.363049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:35.363111 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:35.363213 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:35.363509 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.364339 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.365540 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.365906 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.366031 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.366312 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:35.366927 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:35.367062 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:35.367125 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 20:13:35.367184 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260620 20:13:35.380442 26953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.374533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:46401 (local address 127.24.20.129:43745)
0620 20:13:35.376819 (+ 2286us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:35.376834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:35.376935 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:35.377196 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:35.377199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:35.377206 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:13:35.377553 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.377558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.378558 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.378560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.379161 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.379169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.379753 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.379756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.379851 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.379900 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:13:35.380201 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:13:35.380205 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:13:35.380255 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 20:13:35.380313 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2188,"thread_start_us":102,"threads_started":1}
I20260620 20:13:35.380441 26892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.374627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:43745 (local address 127.24.20.131:46401)
0620 20:13:35.374731 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:35.374736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:35.376917 (+ 2181us) server_negotiation.cc:407] Connection header received
0620 20:13:35.377032 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:35.377035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:35.377089 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:35.377166 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:35.377661 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.378465 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.379295 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.379629 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.379734 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.380037 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:35.380114 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:35.380184 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:35.380267 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 20:13:35.380315 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:13:35.386387 26825 raft_consensus.cc:1261] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Refusing update from remote peer d90b85ee0494454dbfc41e43b6c7b551: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:35.386407 26684 raft_consensus.cc:1261] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Refusing update from remote peer d90b85ee0494454dbfc41e43b6c7b551: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:35.386976 26901 consensus_queue.cc:1035] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:35.387181 26949 consensus_queue.cc:1035] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:35.407612 26877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.404319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:41731 (local address 127.24.20.131:60193)
0620 20:13:35.404419 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:35.404431 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:35.404492 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:35.404999 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:35.405002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:35.405010 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:13:35.405101 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.405107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.405806 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.405811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.406396 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:35.406404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.406943 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.406946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:35.407030 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.407064 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:13:35.407359 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:13:35.407364 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:13:35.407428 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 20:13:35.407504 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 20:13:35.407645 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:35.404683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:60193 (local address 127.24.20.129:41731)
0620 20:13:35.404798 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:35.404802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:35.404817 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:13:35.404857 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:35.404859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:35.404906 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:35.404989 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:35.405209 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.405701 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.406510 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:35.406837 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:35.406948 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:35.407181 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:35.407259 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:35.407319 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:35.407448 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 20:13:35.407503 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 20:13:35.408370 26923 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 20:13:36.967988 26387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:36.968127 26387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:36.970378 26387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb: Sending DeleteTablet for 3 replicas of tablet 42eadd09bf2846699de998531692198a
I20260620 20:13:36.970914 26525 tablet_service.cc:1514] Processing DeleteTablet for tablet 42eadd09bf2846699de998531692198a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:36 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:38500
I20260620 20:13:36.971040 26665 tablet_service.cc:1514] Processing DeleteTablet for tablet 42eadd09bf2846699de998531692198a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:36 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:44000
I20260620 20:13:36.971077 26805 tablet_service.cc:1514] Processing DeleteTablet for tablet 42eadd09bf2846699de998531692198a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:36 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:46486
I20260620 20:13:36.971354 26983 tablet_replica.cc:330] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529: stopping tablet replica
I20260620 20:13:36.971482 26983 raft_consensus.cc:2229] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:36.971508 26982 tablet_replica.cc:330] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551: stopping tablet replica
I20260620 20:13:36.971594 26983 raft_consensus.cc:2258] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:36.971740 26982 raft_consensus.cc:2229] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:13:36.971829 26984 tablet_replica.cc:330] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b: stopping tablet replica
I20260620 20:13:36.971980 26984 raft_consensus.cc:2229] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:36.972095 26984 raft_consensus.cc:2258] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:36.972391 26983 ts_tablet_manager.cc:1905] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:36.972396 26982 raft_consensus.cc:2258] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:36.972751 26984 ts_tablet_manager.cc:1905] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:36.973193 26982 ts_tablet_manager.cc:1905] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:36.975193 26983 ts_tablet_manager.cc:1918] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:13:36.975270 26983 log.cc:1196] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/42eadd09bf2846699de998531692198a
I20260620 20:13:36.975337 26984 ts_tablet_manager.cc:1918] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:13:36.975476 26984 log.cc:1196] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/42eadd09bf2846699de998531692198a
I20260620 20:13:36.975584 26983 ts_tablet_manager.cc:1939] T 42eadd09bf2846699de998531692198a P 9efceca70ade4461b426c1e4a1a72529: Deleting consensus metadata
I20260620 20:13:36.975795 26984 ts_tablet_manager.cc:1939] T 42eadd09bf2846699de998531692198a P f697d6be43894c6f9eb063522e44f51b: Deleting consensus metadata
I20260620 20:13:36.976207 26982 ts_tablet_manager.cc:1918] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:13:36.976284 26982 log.cc:1196] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/wal/wals/42eadd09bf2846699de998531692198a
I20260620 20:13:36.976375 26375 catalog_manager.cc:4841] TS f697d6be43894c6f9eb063522e44f51b (127.24.20.130:38355): tablet 42eadd09bf2846699de998531692198a (table test-table [id=040013c5e0714a65934165c071b118d3]) successfully deleted
I20260620 20:13:36.976490 26373 catalog_manager.cc:4841] TS 9efceca70ade4461b426c1e4a1a72529 (127.24.20.131:46401): tablet 42eadd09bf2846699de998531692198a (table test-table [id=040013c5e0714a65934165c071b118d3]) successfully deleted
I20260620 20:13:36.976647 26982 ts_tablet_manager.cc:1939] T 42eadd09bf2846699de998531692198a P d90b85ee0494454dbfc41e43b6c7b551: Deleting consensus metadata
I20260620 20:13:36.977386 26375 catalog_manager.cc:4841] TS d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129:41731): tablet 42eadd09bf2846699de998531692198a (table test-table [id=040013c5e0714a65934165c071b118d3]) successfully deleted
Jun 20 20:13:36 dist-test-slave-pflr krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986416, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:13:36 dist-test-slave-pflr krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986416, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:36.999783 26992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:36.991484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.24.20.190:33721)
0620 20:13:36.991747 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:36.991756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:36.991850 (+ 94us) server_negotiation.cc:407] Connection header received
0620 20:13:36.992013 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:36.992016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:36.992069 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:36.992171 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:36.992891 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:36.993439 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:36.994272 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:36.994502 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:36.994611 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:36.996888 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:36.996892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:36.996895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:36.996923 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:36.998192 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:36.998589 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:36.998594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:36.998596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:36.998679 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:36.998947 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:36.998952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:36.998953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:36.999133 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:36.999271 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:36.999491 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 20:13:36.999611 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
I20260620 20:13:37.002086 26387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:13:37.002462 26387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:37.006772 26525 tablet_service.cc:1467] Processing CreateTablet for tablet c9f63fe6c76641f28e72111f4a6ab110 (DEFAULT_TABLE table=test-table [id=2e8eb66b184e4cc0bced84c40a95d920]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:37.006821 26665 tablet_service.cc:1467] Processing CreateTablet for tablet c9f63fe6c76641f28e72111f4a6ab110 (DEFAULT_TABLE table=test-table [id=2e8eb66b184e4cc0bced84c40a95d920]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:37.006917 26805 tablet_service.cc:1467] Processing CreateTablet for tablet c9f63fe6c76641f28e72111f4a6ab110 (DEFAULT_TABLE table=test-table [id=2e8eb66b184e4cc0bced84c40a95d920]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:37.007071 26525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f63fe6c76641f28e72111f4a6ab110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:37.007117 26665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f63fe6c76641f28e72111f4a6ab110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:37.007176 26805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f63fe6c76641f28e72111f4a6ab110. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:37.009115 26993 tablet_bootstrap.cc:492] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b: Bootstrap starting.
I20260620 20:13:37.009459 26995 tablet_bootstrap.cc:492] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551: Bootstrap starting.
I20260620 20:13:37.009671 26994 tablet_bootstrap.cc:492] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529: Bootstrap starting.
I20260620 20:13:37.010044 26993 tablet_bootstrap.cc:654] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:37.010921 26995 tablet_bootstrap.cc:654] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:37.011159 26993 tablet_bootstrap.cc:492] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b: No bootstrap required, opened a new log
I20260620 20:13:37.011250 26993 ts_tablet_manager.cc:1397] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:37.011674 26993 raft_consensus.cc:348] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.011791 26993 raft_consensus.cc:374] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:37.011840 26993 raft_consensus.cc:729] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f697d6be43894c6f9eb063522e44f51b, State: Initialized, Role: FOLLOWER
I20260620 20:13:37.011916 26994 tablet_bootstrap.cc:654] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:37.012008 26993 consensus_queue.cc:260] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.012092 26995 tablet_bootstrap.cc:492] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551: No bootstrap required, opened a new log
I20260620 20:13:37.012234 26995 ts_tablet_manager.cc:1397] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:13:37.012384 26993 ts_tablet_manager.cc:1428] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:37.012735 26995 raft_consensus.cc:348] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.012861 26995 raft_consensus.cc:374] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:37.012900 26995 raft_consensus.cc:729] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90b85ee0494454dbfc41e43b6c7b551, State: Initialized, Role: FOLLOWER
I20260620 20:13:37.013062 26994 tablet_bootstrap.cc:492] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529: No bootstrap required, opened a new log
I20260620 20:13:37.013033 26995 consensus_queue.cc:260] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.013160 26994 ts_tablet_manager.cc:1397] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 20:13:37.013502 26994 raft_consensus.cc:348] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.013615 26994 raft_consensus.cc:374] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:37.013646 26994 raft_consensus.cc:729] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efceca70ade4461b426c1e4a1a72529, State: Initialized, Role: FOLLOWER
I20260620 20:13:37.013752 26995 ts_tablet_manager.cc:1428] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:37.013767 26994 consensus_queue.cc:260] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.014027 26994 ts_tablet_manager.cc:1428] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:37.033425 26998 raft_consensus.cc:482] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:37.033586 26998 raft_consensus.cc:504] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.033983 26998 leader_election.cc:288] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129:41731), 9efceca70ade4461b426c1e4a1a72529 (127.24.20.131:46401)
I20260620 20:13:37.034328 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f63fe6c76641f28e72111f4a6ab110" candidate_uuid: "f697d6be43894c6f9eb063522e44f51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90b85ee0494454dbfc41e43b6c7b551" is_pre_election: true
I20260620 20:13:37.034328 26825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f63fe6c76641f28e72111f4a6ab110" candidate_uuid: "f697d6be43894c6f9eb063522e44f51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efceca70ade4461b426c1e4a1a72529" is_pre_election: true
I20260620 20:13:37.034467 26545 raft_consensus.cc:2454] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f697d6be43894c6f9eb063522e44f51b in term 0.
I20260620 20:13:37.034483 26825 raft_consensus.cc:2454] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f697d6be43894c6f9eb063522e44f51b in term 0.
I20260620 20:13:37.034830 26619 leader_election.cc:302] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9efceca70ade4461b426c1e4a1a72529, d90b85ee0494454dbfc41e43b6c7b551, f697d6be43894c6f9eb063522e44f51b; no voters:
I20260620 20:13:37.035001 26998 raft_consensus.cc:2781] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:37.035064 26998 raft_consensus.cc:482] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:37.035094 26998 raft_consensus.cc:3037] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:37.036007 26998 raft_consensus.cc:504] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.036370 26998 leader_election.cc:288] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [CANDIDATE]: Term 1 election: Requested vote from peers d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129:41731), 9efceca70ade4461b426c1e4a1a72529 (127.24.20.131:46401)
I20260620 20:13:37.036715 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f63fe6c76641f28e72111f4a6ab110" candidate_uuid: "f697d6be43894c6f9eb063522e44f51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90b85ee0494454dbfc41e43b6c7b551"
I20260620 20:13:37.036777 26825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f63fe6c76641f28e72111f4a6ab110" candidate_uuid: "f697d6be43894c6f9eb063522e44f51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efceca70ade4461b426c1e4a1a72529"
I20260620 20:13:37.036832 26545 raft_consensus.cc:3037] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:37.036865 26825 raft_consensus.cc:3037] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:37.037891 26825 raft_consensus.cc:2454] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f697d6be43894c6f9eb063522e44f51b in term 1.
I20260620 20:13:37.037891 26545 raft_consensus.cc:2454] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f697d6be43894c6f9eb063522e44f51b in term 1.
I20260620 20:13:37.038156 26619 leader_election.cc:302] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d90b85ee0494454dbfc41e43b6c7b551, f697d6be43894c6f9eb063522e44f51b; no voters:
I20260620 20:13:37.038311 26998 raft_consensus.cc:2781] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:37.038532 26998 raft_consensus.cc:686] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [term 1 LEADER]: Becoming Leader. State: Replica: f697d6be43894c6f9eb063522e44f51b, State: Running, Role: LEADER
I20260620 20:13:37.038683 26998 consensus_queue.cc:237] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.040414 26386 catalog_manager.cc:5495] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b reported cstate change: term changed from 0 to 1, leader changed from <none> to f697d6be43894c6f9eb063522e44f51b (127.24.20.130). New cstate: current_term: 1 leader_uuid: "f697d6be43894c6f9eb063522e44f51b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } health_report { overall_health: UNKNOWN } } }
I20260620 20:13:37.059543 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.056673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.24.20.130:38355)
0620 20:13:37.056908 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.056915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.056927 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:13:37.056967 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.056969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.057007 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.057088 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:37.057313 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.057863 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.058681 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.058896 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.058980 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.059199 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:37.059263 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:37.059316 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.059390 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.059434 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":110,"threads_started":1}
I20260620 20:13:37.065241 26825 raft_consensus.cc:1261] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Refusing update from remote peer f697d6be43894c6f9eb063522e44f51b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:37.065290 26545 raft_consensus.cc:1261] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Refusing update from remote peer f697d6be43894c6f9eb063522e44f51b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:37.065680 27000 consensus_queue.cc:1035] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:37.065824 26998 consensus_queue.cc:1035] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:37.095961 26387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:37.096092 26387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:37.098018 26387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb: Sending DeleteTablet for 3 replicas of tablet c9f63fe6c76641f28e72111f4a6ab110
I20260620 20:13:37.098541 26665 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f63fe6c76641f28e72111f4a6ab110 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:37 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:44000
I20260620 20:13:37.098591 26525 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f63fe6c76641f28e72111f4a6ab110 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:37 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:38500
I20260620 20:13:37.098740 26984 tablet_replica.cc:330] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b: stopping tablet replica
I20260620 20:13:37.098757 26982 tablet_replica.cc:330] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551: stopping tablet replica
I20260620 20:13:37.098830 26984 raft_consensus.cc:2229] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:13:37.098869 26982 raft_consensus.cc:2229] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:37.098905 26805 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f63fe6c76641f28e72111f4a6ab110 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:37 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:46486
I20260620 20:13:37.098973 26982 raft_consensus.cc:2258] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:37.099015 26984 raft_consensus.cc:2258] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:37.099053 26983 tablet_replica.cc:330] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529: stopping tablet replica
I20260620 20:13:37.099161 26983 raft_consensus.cc:2229] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:37.099258 26983 raft_consensus.cc:2258] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:37.099224 26387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 20:13:37.099583 26984 ts_tablet_manager.cc:1905] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 20:13:37.099601 26387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:37.099960 26983 ts_tablet_manager.cc:1905] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:37.099960 26982 ts_tablet_manager.cc:1905] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:37.102329 26983 ts_tablet_manager.cc:1918] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:13:37.102394 26983 log.cc:1196] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/c9f63fe6c76641f28e72111f4a6ab110
I20260620 20:13:37.102399 26982 ts_tablet_manager.cc:1918] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:13:37.102478 26982 log.cc:1196] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-0/wal/wals/c9f63fe6c76641f28e72111f4a6ab110
I20260620 20:13:37.102686 26983 ts_tablet_manager.cc:1939] T c9f63fe6c76641f28e72111f4a6ab110 P 9efceca70ade4461b426c1e4a1a72529: Deleting consensus metadata
I20260620 20:13:37.102753 26982 ts_tablet_manager.cc:1939] T c9f63fe6c76641f28e72111f4a6ab110 P d90b85ee0494454dbfc41e43b6c7b551: Deleting consensus metadata
I20260620 20:13:37.103158 26984 ts_tablet_manager.cc:1918] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:13:37.103227 26984 log.cc:1196] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/c9f63fe6c76641f28e72111f4a6ab110
I20260620 20:13:37.103276 26375 catalog_manager.cc:4841] TS d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129:41731): tablet c9f63fe6c76641f28e72111f4a6ab110 (table test-table [id=2e8eb66b184e4cc0bced84c40a95d920]) successfully deleted
I20260620 20:13:37.103492 26984 ts_tablet_manager.cc:1939] T c9f63fe6c76641f28e72111f4a6ab110 P f697d6be43894c6f9eb063522e44f51b: Deleting consensus metadata
I20260620 20:13:37.103544 26373 catalog_manager.cc:4841] TS 9efceca70ade4461b426c1e4a1a72529 (127.24.20.131:46401): tablet c9f63fe6c76641f28e72111f4a6ab110 (table test-table [id=2e8eb66b184e4cc0bced84c40a95d920]) successfully deleted
I20260620 20:13:37.104039 26375 catalog_manager.cc:4841] TS f697d6be43894c6f9eb063522e44f51b (127.24.20.130:38355): tablet c9f63fe6c76641f28e72111f4a6ab110 (table test-table [id=2e8eb66b184e4cc0bced84c40a95d920]) successfully deleted
I20260620 20:13:37.104667 26525 tablet_service.cc:1467] Processing CreateTablet for tablet 86e2ca7f0e8147fb901ea9c2b8fa6f44 (DEFAULT_TABLE table=test-table [id=dead36922b1c47d28604f795bd2bbf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:37.104696 26665 tablet_service.cc:1467] Processing CreateTablet for tablet 86e2ca7f0e8147fb901ea9c2b8fa6f44 (DEFAULT_TABLE table=test-table [id=dead36922b1c47d28604f795bd2bbf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:37.104902 26525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e2ca7f0e8147fb901ea9c2b8fa6f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:37.104861 26805 tablet_service.cc:1467] Processing CreateTablet for tablet 86e2ca7f0e8147fb901ea9c2b8fa6f44 (DEFAULT_TABLE table=test-table [id=dead36922b1c47d28604f795bd2bbf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:37.104923 26665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e2ca7f0e8147fb901ea9c2b8fa6f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:37.105058 26805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86e2ca7f0e8147fb901ea9c2b8fa6f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:37.106493 26993 tablet_bootstrap.cc:492] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b: Bootstrap starting.
I20260620 20:13:37.106977 26994 tablet_bootstrap.cc:492] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529: Bootstrap starting.
I20260620 20:13:37.107270 26993 tablet_bootstrap.cc:654] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:37.107796 26994 tablet_bootstrap.cc:654] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:37.108326 26993 tablet_bootstrap.cc:492] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b: No bootstrap required, opened a new log
I20260620 20:13:37.108347 26995 tablet_bootstrap.cc:492] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551: Bootstrap starting.
I20260620 20:13:37.108410 26993 ts_tablet_manager.cc:1397] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:37.108853 26994 tablet_bootstrap.cc:492] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529: No bootstrap required, opened a new log
I20260620 20:13:37.108798 26993 raft_consensus.cc:348] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.108914 26993 raft_consensus.cc:374] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:37.108950 26994 ts_tablet_manager.cc:1397] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:37.108956 26993 raft_consensus.cc:729] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f697d6be43894c6f9eb063522e44f51b, State: Initialized, Role: FOLLOWER
I20260620 20:13:37.109114 26995 tablet_bootstrap.cc:654] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:37.109086 26993 consensus_queue.cc:260] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.109352 26993 ts_tablet_manager.cc:1428] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:37.109364 26994 raft_consensus.cc:348] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.109486 26994 raft_consensus.cc:374] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:37.109529 26994 raft_consensus.cc:729] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efceca70ade4461b426c1e4a1a72529, State: Initialized, Role: FOLLOWER
I20260620 20:13:37.109655 26994 consensus_queue.cc:260] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.109915 26994 ts_tablet_manager.cc:1428] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 20:13:37.111349 26995 tablet_bootstrap.cc:492] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551: No bootstrap required, opened a new log
I20260620 20:13:37.111430 26995 ts_tablet_manager.cc:1397] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 20:13:37.111837 26995 raft_consensus.cc:348] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.111953 26995 raft_consensus.cc:374] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:37.111990 26995 raft_consensus.cc:729] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90b85ee0494454dbfc41e43b6c7b551, State: Initialized, Role: FOLLOWER
I20260620 20:13:37.112120 26995 consensus_queue.cc:260] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.112381 26995 ts_tablet_manager.cc:1428] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 20:13:37.126714 26968 raft_consensus.cc:482] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:37.126821 26968 raft_consensus.cc:504] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.127223 26968 leader_election.cc:288] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129:41731), f697d6be43894c6f9eb063522e44f51b (127.24.20.130:38355)
I20260620 20:13:37.127589 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e2ca7f0e8147fb901ea9c2b8fa6f44" candidate_uuid: "9efceca70ade4461b426c1e4a1a72529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90b85ee0494454dbfc41e43b6c7b551" is_pre_election: true
I20260620 20:13:37.127626 26684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e2ca7f0e8147fb901ea9c2b8fa6f44" candidate_uuid: "9efceca70ade4461b426c1e4a1a72529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f697d6be43894c6f9eb063522e44f51b" is_pre_election: true
I20260620 20:13:37.127751 26545 raft_consensus.cc:2454] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efceca70ade4461b426c1e4a1a72529 in term 0.
I20260620 20:13:37.127751 26684 raft_consensus.cc:2454] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efceca70ade4461b426c1e4a1a72529 in term 0.
I20260620 20:13:37.128036 26759 leader_election.cc:302] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9efceca70ade4461b426c1e4a1a72529, f697d6be43894c6f9eb063522e44f51b; no voters:
I20260620 20:13:37.128194 26968 raft_consensus.cc:2781] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:37.128254 26968 raft_consensus.cc:482] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:37.128297 26968 raft_consensus.cc:3037] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:37.129319 26968 raft_consensus.cc:504] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.129632 26968 leader_election.cc:288] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [CANDIDATE]: Term 1 election: Requested vote from peers d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129:41731), f697d6be43894c6f9eb063522e44f51b (127.24.20.130:38355)
I20260620 20:13:37.129896 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e2ca7f0e8147fb901ea9c2b8fa6f44" candidate_uuid: "9efceca70ade4461b426c1e4a1a72529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90b85ee0494454dbfc41e43b6c7b551"
I20260620 20:13:37.130013 26545 raft_consensus.cc:3037] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:37.129973 26684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86e2ca7f0e8147fb901ea9c2b8fa6f44" candidate_uuid: "9efceca70ade4461b426c1e4a1a72529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f697d6be43894c6f9eb063522e44f51b"
I20260620 20:13:37.130113 26684 raft_consensus.cc:3037] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:37.130887 26545 raft_consensus.cc:2454] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efceca70ade4461b426c1e4a1a72529 in term 1.
I20260620 20:13:37.131112 26684 raft_consensus.cc:2454] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efceca70ade4461b426c1e4a1a72529 in term 1.
I20260620 20:13:37.131173 26759 leader_election.cc:302] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9efceca70ade4461b426c1e4a1a72529, d90b85ee0494454dbfc41e43b6c7b551; no voters:
I20260620 20:13:37.131304 26968 raft_consensus.cc:2781] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:37.131390 26968 raft_consensus.cc:686] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [term 1 LEADER]: Becoming Leader. State: Replica: 9efceca70ade4461b426c1e4a1a72529, State: Running, Role: LEADER
I20260620 20:13:37.131524 26968 consensus_queue.cc:237] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } }
I20260620 20:13:37.132954 26387 catalog_manager.cc:5495] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9efceca70ade4461b426c1e4a1a72529 (127.24.20.131). New cstate: current_term: 1 leader_uuid: "9efceca70ade4461b426c1e4a1a72529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9efceca70ade4461b426c1e4a1a72529" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46401 } health_report { overall_health: HEALTHY } } }
I20260620 20:13:37.147521 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.144389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.24.20.131:46401)
0620 20:13:37.144703 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.144712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.144727 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:13:37.144761 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.144763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.144799 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.144886 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:37.145118 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.145663 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.146464 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.146720 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.146825 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.147054 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:37.147131 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:37.147207 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.147321 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.147371 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":124,"threads_started":1}
I20260620 20:13:37.155256 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.152543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:55899 (local address 127.24.20.131:46401)
0620 20:13:37.152704 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:37.152709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:37.152725 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:13:37.152822 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:37.152825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:37.152871 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:37.152949 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:37.153124 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.153568 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.154392 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.154587 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.154671 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.154861 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:37.154915 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:37.155010 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:37.155089 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 20:13:37.155130 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 20:13:37.155293 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:37.152394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:46401 (local address 127.24.20.131:55899)
0620 20:13:37.152675 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:37.152687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:37.152757 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:37.152957 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:37.152958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:37.152964 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:13:37.153046 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.153050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.153660 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.153662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.154307 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:37.154316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:37.154682 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:37.154684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:37.154741 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:37.154762 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:13:37.155012 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:13:37.155019 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:13:37.155082 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 20:13:37.155176 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260620 20:13:37.160385 26684 raft_consensus.cc:1261] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Refusing update from remote peer 9efceca70ade4461b426c1e4a1a72529: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:37.160395 26545 raft_consensus.cc:1261] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Refusing update from remote peer 9efceca70ade4461b426c1e4a1a72529: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:37.160751 26968 consensus_queue.cc:1035] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f697d6be43894c6f9eb063522e44f51b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:37.160887 26963 consensus_queue.cc:1035] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d90b85ee0494454dbfc41e43b6c7b551" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:13:37.174646 26960 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 20:13:38.726585 26386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:38.726732 26386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:38.728940 26386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35cc233f12ab4f2d977c09d8abf5c9fb: Sending DeleteTablet for 3 replicas of tablet 86e2ca7f0e8147fb901ea9c2b8fa6f44
I20260620 20:13:38.729457 26525 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e2ca7f0e8147fb901ea9c2b8fa6f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:38 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:38500
I20260620 20:13:38.729501 26805 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e2ca7f0e8147fb901ea9c2b8fa6f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:38 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:46486
I20260620 20:13:38.729552 26665 tablet_service.cc:1514] Processing DeleteTablet for tablet 86e2ca7f0e8147fb901ea9c2b8fa6f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:38 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:44000
I20260620 20:13:38.729791 27041 tablet_replica.cc:330] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551: stopping tablet replica
I20260620 20:13:38.729826 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 26460
I20260620 20:13:38.729826 27043 tablet_replica.cc:330] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529: stopping tablet replica
I20260620 20:13:38.729944 27041 raft_consensus.cc:2229] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:38.730058 27041 raft_consensus.cc:2258] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P d90b85ee0494454dbfc41e43b6c7b551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:38.729948 27043 raft_consensus.cc:2229] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:13:38.730625 27042 tablet_replica.cc:330] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b: stopping tablet replica
I20260620 20:13:38.730721 27043 raft_consensus.cc:2258] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:38.730988 27042 raft_consensus.cc:2229] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:38.731160 27042 raft_consensus.cc:2258] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:38.731349 27043 ts_tablet_manager.cc:1905] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:38.731834 27042 ts_tablet_manager.cc:1905] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:38.734102 27043 ts_tablet_manager.cc:1918] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:13:38.734184 27043 log.cc:1196] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/86e2ca7f0e8147fb901ea9c2b8fa6f44
I20260620 20:13:38.734480 27043 ts_tablet_manager.cc:1939] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P 9efceca70ade4461b426c1e4a1a72529: Deleting consensus metadata
I20260620 20:13:38.734484 27042 ts_tablet_manager.cc:1918] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:13:38.734545 27042 log.cc:1196] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/86e2ca7f0e8147fb901ea9c2b8fa6f44
I20260620 20:13:38.734795 27042 ts_tablet_manager.cc:1939] T 86e2ca7f0e8147fb901ea9c2b8fa6f44 P f697d6be43894c6f9eb063522e44f51b: Deleting consensus metadata
I20260620 20:13:38.735203 26373 catalog_manager.cc:4841] TS 9efceca70ade4461b426c1e4a1a72529 (127.24.20.131:46401): tablet 86e2ca7f0e8147fb901ea9c2b8fa6f44 (table test-table [id=dead36922b1c47d28604f795bd2bbf46]) successfully deleted
I20260620 20:13:38.735457 26375 catalog_manager.cc:4841] TS f697d6be43894c6f9eb063522e44f51b (127.24.20.130:38355): tablet 86e2ca7f0e8147fb901ea9c2b8fa6f44 (table test-table [id=dead36922b1c47d28604f795bd2bbf46]) successfully deleted
W20260620 20:13:38.738106 26375 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:41731 (error 108)
I20260620 20:13:38.738278 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 26600
I20260620 20:13:38.739377 27045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:38.738905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:41731 (local address 127.0.0.1:54718)
0620 20:13:38.739159 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:38.739249 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:41731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
W20260620 20:13:38.739518 26375 catalog_manager.cc:4584] TS d90b85ee0494454dbfc41e43b6c7b551 (127.24.20.129:41731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86e2ca7f0e8147fb901ea9c2b8fa6f44: Network error: Client connection negotiation failed: client connection to 127.24.20.129:41731: connect: Connection refused (error 111)
I20260620 20:13:38.746024 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 26740
I20260620 20:13:38.752424 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 26356
2026-06-20T20:13:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5949 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:13:38 dist-test-slave-pflr krb5kdc[27048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:13:38 dist-test-slave-pflr krb5kdc[27048](info): set up 2 sockets
Jun 20 20:13:38 dist-test-slave-pflr krb5kdc[27048](info): commencing operation
krb5kdc: starting...
W20260620 20:13:39.796905 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:13:39 dist-test-slave-pflr krb5kdc[27048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:13:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:13:39.911716 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:44999
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:44999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:40.017020 27061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:40.017293 27061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:40.017364 27061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:40.021257 27061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:40.021335 27061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:40.021374 27061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:40.021431 27061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:40.021467 27061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:40.026602 27061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:44999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:44999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:40.027556 27061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:40.028507 27061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:40.036654 27066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:40.036644 27067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:40.036710 27061 server_base.cc:1034] running on GCE node
W20260620 20:13:40.036641 27069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:40.037465 27061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:40.038504 27061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:40.039665 27061 hybrid_clock.cc:648] HybridClock initialized: now 1781986420039651 us; error 24 us; skew 500 ppm
Jun 20 20:13:40 dist-test-slave-pflr krb5kdc[27048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986420, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:40.042483 27061 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:13:40.043309 27061 webserver.cc:460] Webserver started at http://127.24.20.190:43321/ using document root <none> and password file <none>
I20260620 20:13:40.043848 27061 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:40.043936 27061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:40.044174 27061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:40.046175 27061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "2da41bb4c3ae4a3193c6d9e264c33f47"
format_stamp: "Formatted at 2026-06-20 20:13:40 on dist-test-slave-pflr"
server_key: "c2c2eab711ded5622a3deb3c0160f31a"
server_key_iv: "534fe79245b89101bb3d7198e18939f0"
server_key_version: "encryptionkey@0"
I20260620 20:13:40.046718 27061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "2da41bb4c3ae4a3193c6d9e264c33f47"
format_stamp: "Formatted at 2026-06-20 20:13:40 on dist-test-slave-pflr"
server_key: "c2c2eab711ded5622a3deb3c0160f31a"
server_key_iv: "534fe79245b89101bb3d7198e18939f0"
server_key_version: "encryptionkey@0"
I20260620 20:13:40.050269 27061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 20:13:40.052543 27076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:40.053751 27061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:13:40.053886 27061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "2da41bb4c3ae4a3193c6d9e264c33f47"
format_stamp: "Formatted at 2026-06-20 20:13:40 on dist-test-slave-pflr"
server_key: "c2c2eab711ded5622a3deb3c0160f31a"
server_key_iv: "534fe79245b89101bb3d7198e18939f0"
server_key_version: "encryptionkey@0"
I20260620 20:13:40.053993 27061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:40.066116 27061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:40.069386 27061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:40.069590 27061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:40.076340 27061 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:44999
I20260620 20:13:40.076406 27128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:44999 every 8 connection(s)
I20260620 20:13:40.077389 27061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:40.080148 27129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:40.085377 27129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47: Bootstrap starting.
I20260620 20:13:40.086802 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 27061
I20260620 20:13:40.086887 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:40.087172 24658 external_mini_cluster.cc:1428] Setting key e8e8c09d3bf4ff480017c1162b4ad930
I20260620 20:13:40.087869 27129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:40.088732 27129 log.cc:824] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:40.090793 27129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47: No bootstrap required, opened a new log
I20260620 20:13:40.095000 27129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da41bb4c3ae4a3193c6d9e264c33f47" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44999 } }
I20260620 20:13:40.095204 27129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:40.095263 27129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da41bb4c3ae4a3193c6d9e264c33f47, State: Initialized, Role: FOLLOWER
I20260620 20:13:40.095697 27129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da41bb4c3ae4a3193c6d9e264c33f47" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44999 } }
I20260620 20:13:40.095811 27129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:40.095894 27129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:40.095992 27129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 20:13:40 dist-test-slave-pflr krb5kdc[27048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:40.096935 27129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da41bb4c3ae4a3193c6d9e264c33f47" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44999 } }
I20260620 20:13:40.097311 27129 leader_election.cc:302] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2da41bb4c3ae4a3193c6d9e264c33f47; no voters:
I20260620 20:13:40.097641 27129 leader_election.cc:288] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:40.097712 27134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:40.097903 27134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [term 1 LEADER]: Becoming Leader. State: Replica: 2da41bb4c3ae4a3193c6d9e264c33f47, State: Running, Role: LEADER
I20260620 20:13:40.098191 27134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da41bb4c3ae4a3193c6d9e264c33f47" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44999 } }
I20260620 20:13:40.098865 27129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:40.101442 27136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2da41bb4c3ae4a3193c6d9e264c33f47. Latest consensus state: current_term: 1 leader_uuid: "2da41bb4c3ae4a3193c6d9e264c33f47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da41bb4c3ae4a3193c6d9e264c33f47" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44999 } } }
I20260620 20:13:40.101560 27136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:40.101665 27135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2da41bb4c3ae4a3193c6d9e264c33f47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da41bb4c3ae4a3193c6d9e264c33f47" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44999 } } }
I20260620 20:13:40.103204 27135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:40.102878 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:40.088411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.24.20.190:44999)
0620 20:13:40.088936 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:40.088955 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:40.088996 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:13:40.089785 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:40.089807 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:40.090180 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:40.090521 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:40.091294 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.092825 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.093682 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.094413 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.094570 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:40.097430 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:40.097438 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:40.097458 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:40.097487 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:40.099815 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:40.100318 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:40.100323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:40.100329 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:40.100459 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:40.100740 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:40.100743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:40.100745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:40.101276 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:40.101491 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:40.101951 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 20:13:40.102205 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":163,"threads_started":1}
W20260620 20:13:40.104007 27149 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:13:40.104079 27149 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:13:40.104161 27151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:40.104983 27151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:40.110244 27151 catalog_manager.cc:1347] Generated new cluster ID: 395cbea0902b44b5b644156ef982109d
I20260620 20:13:40.110311 27151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:40.151875 27151 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:40.152908 27151 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:40.158748 27151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2da41bb4c3ae4a3193c6d9e264c33f47: Generated new TSK 0
I20260620 20:13:40.159412 27151 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:13:40.195163 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44999
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:40.326720 27155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:40.326952 27155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:40.326992 27155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:40.330797 27155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:40.330842 27155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:40.330895 27155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:13:40.335760 27155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:40.336638 27155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:40.337519 27155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:40.346490 27160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:40.346517 27163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:40.346537 27155 server_base.cc:1034] running on GCE node
W20260620 20:13:40.346666 27161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:40.347020 27155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:40.347656 27155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:40.348829 27155 hybrid_clock.cc:648] HybridClock initialized: now 1781986420348804 us; error 44 us; skew 500 ppm
Jun 20 20:13:40 dist-test-slave-pflr krb5kdc[27048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986420, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:40.351406 27155 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:13:40.352226 27155 webserver.cc:460] Webserver started at http://127.24.20.129:44223/ using document root <none> and password file <none>
I20260620 20:13:40.352808 27155 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:40.352895 27155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:40.353135 27155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:40.355019 27155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "a105c9e48e3f4ff8bcc797c4e736f9f1"
format_stamp: "Formatted at 2026-06-20 20:13:40 on dist-test-slave-pflr"
server_key: "d07b54e4da0b48bc9559b609229a2dd2"
server_key_iv: "47562f73083b4214702f25edcb364c41"
server_key_version: "encryptionkey@0"
I20260620 20:13:40.355562 27155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "a105c9e48e3f4ff8bcc797c4e736f9f1"
format_stamp: "Formatted at 2026-06-20 20:13:40 on dist-test-slave-pflr"
server_key: "d07b54e4da0b48bc9559b609229a2dd2"
server_key_iv: "47562f73083b4214702f25edcb364c41"
server_key_version: "encryptionkey@0"
I20260620 20:13:40.359190 27155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:13:40.361557 27170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:40.362726 27155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:40.362856 27155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "a105c9e48e3f4ff8bcc797c4e736f9f1"
format_stamp: "Formatted at 2026-06-20 20:13:40 on dist-test-slave-pflr"
server_key: "d07b54e4da0b48bc9559b609229a2dd2"
server_key_iv: "47562f73083b4214702f25edcb364c41"
server_key_version: "encryptionkey@0"
I20260620 20:13:40.362964 27155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:40.380357 27155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:40.383252 27155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:40.383452 27155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:40.384024 27155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:40.384871 27155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:40.384940 27155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:40.385010 27155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:40.385052 27155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:40.393734 27155 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:40283
I20260620 20:13:40.393757 27283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:40283 every 8 connection(s)
I20260620 20:13:40.394565 27155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:40.401739 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 27155
I20260620 20:13:40.401872 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:13:40.402180 24658 external_mini_cluster.cc:1428] Setting key fa517ecef0216296bf739c2308b007f8
Jun 20 20:13:40 dist-test-slave-pflr krb5kdc[27048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986420, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:40.408221 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:40.396308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:43213 (local address 127.24.20.190:44999)
0620 20:13:40.396436 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:40.396440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:40.397231 (+ 791us) server_negotiation.cc:407] Connection header received
0620 20:13:40.398559 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:40.398562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:40.398627 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:40.398720 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:40.399953 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.400541 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.401762 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.402075 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.402165 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:40.404736 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:40.404739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:40.404742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:40.404766 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:40.406246 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:40.406752 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:40.406757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:40.406759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:40.406838 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:40.407168 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:40.407173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:40.407175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:40.407346 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:40.407452 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:40.407919 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 20:13:40.408037 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:13:40.409008 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:40.396675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44999 (local address 127.24.20.129:43213)
0620 20:13:40.397088 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:40.397128 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:40.398339 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:40.398882 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:40.398888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:40.399225 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:40.399791 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:40.399804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.400699 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.400702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:40.401645 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:40.401656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.402222 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.402227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:40.402435 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:40.403065 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:40.403080 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:40.404479 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:40.406376 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:40.406382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:40.406394 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:40.406651 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:40.406936 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:40.406939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:40.406941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:40.407073 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:40.407449 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:40.407456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:40.407691 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 20:13:40.407871 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 20:13:40.408164 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":115,"threads_started":1}
I20260620 20:13:40.410154 27284 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44999
I20260620 20:13:40.410396 27284 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:13:40.410930 27284 heartbeater.cc:511] Master 127.24.20.190:44999 requested a full tablet report, sending...
I20260620 20:13:40.412415 27093 ts_manager.cc:194] Registered new tserver with Master: a105c9e48e3f4ff8bcc797c4e736f9f1 (127.24.20.129:40283)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
I20260620 20:13:40.413676 27093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:43213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:13:40.427754 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44999
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:40.533597 27289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:40.533852 27289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:40.533895 27289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:40.537868 27289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:40.537925 27289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:40.538013 27289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:13:40.543275 27289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:40.544261 27289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:40.545291 27289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:40.554409 27297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:40.554492 27289 server_base.cc:1034] running on GCE node
W20260620 20:13:40.554420 27295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:40.554569 27294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:40.554993 27289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:40.555600 27289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:40.556788 27289 hybrid_clock.cc:648] HybridClock initialized: now 1781986420556771 us; error 24 us; skew 500 ppm
Jun 20 20:13:40 dist-test-slave-pflr krb5kdc[27048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986420, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:40.559367 27289 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:13:40.560175 27289 webserver.cc:460] Webserver started at http://127.24.20.130:39135/ using document root <none> and password file <none>
I20260620 20:13:40.560750 27289 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:40.560837 27289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:40.561057 27289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:40.562923 27289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "d5f92c9c0da7491bb79c03d4bcd85760"
format_stamp: "Formatted at 2026-06-20 20:13:40 on dist-test-slave-pflr"
server_key: "0dc4bb73428bcffffab8673b78ffe5fe"
server_key_iv: "09c1e67947dc07e84f3cad8bdde3a4e4"
server_key_version: "encryptionkey@0"
I20260620 20:13:40.563468 27289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "d5f92c9c0da7491bb79c03d4bcd85760"
format_stamp: "Formatted at 2026-06-20 20:13:40 on dist-test-slave-pflr"
server_key: "0dc4bb73428bcffffab8673b78ffe5fe"
server_key_iv: "09c1e67947dc07e84f3cad8bdde3a4e4"
server_key_version: "encryptionkey@0"
I20260620 20:13:40.567022 27289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:13:40.569422 27304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:40.570557 27289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:40.570792 27289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "d5f92c9c0da7491bb79c03d4bcd85760"
format_stamp: "Formatted at 2026-06-20 20:13:40 on dist-test-slave-pflr"
server_key: "0dc4bb73428bcffffab8673b78ffe5fe"
server_key_iv: "09c1e67947dc07e84f3cad8bdde3a4e4"
server_key_version: "encryptionkey@0"
I20260620 20:13:40.570914 27289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:40.581885 27289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:40.584422 27289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:40.584658 27289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:40.585191 27289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:40.585919 27289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:40.585987 27289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:40.586056 27289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:40.586109 27289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:40.594599 27289 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:39445
I20260620 20:13:40.594624 27417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:39445 every 8 connection(s)
I20260620 20:13:40.595398 27289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:40.603178 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 27289
I20260620 20:13:40.603296 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:13:40.603564 24658 external_mini_cluster.cc:1428] Setting key 27ee915968a1e5d5d0924d1152d5cfd4
Jun 20 20:13:40 dist-test-slave-pflr krb5kdc[27048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986420, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:40.609608 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:40.597075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:38973 (local address 127.24.20.190:44999)
0620 20:13:40.597234 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:40.597238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:40.597988 (+ 750us) server_negotiation.cc:407] Connection header received
0620 20:13:40.599482 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:40.599486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:40.599562 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:40.599671 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:40.601002 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.601749 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.603047 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.603314 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.603416 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:40.605933 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:40.605940 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:40.605946 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:40.605980 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:40.607468 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:40.608046 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:40.608049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:40.608050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:40.608118 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:40.608480 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:40.608482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:40.608483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:40.608716 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:40.608820 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:40.609407 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 20:13:40.609498 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:13:40.610797 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:40.597433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44999 (local address 127.24.20.130:38973)
0620 20:13:40.597850 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:40.597888 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:40.599182 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:40.599896 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:40.599903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:40.600230 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:40.600839 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:40.600860 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.601881 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.601886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:40.602920 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:40.602930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.603552 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.603555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:40.603742 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:40.604284 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:40.604302 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:40.605781 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:40.607578 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:40.607587 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:40.607603 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:40.607944 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:40.608204 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:40.608210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:40.608215 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:40.608387 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:40.608823 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:40.608830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:40.609157 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 20:13:40.609369 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 20:13:40.609704 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260620 20:13:40.611907 27418 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44999
I20260620 20:13:40.612151 27418 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:40.612730 27418 heartbeater.cc:511] Master 127.24.20.190:44999 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
I20260620 20:13:40.613742 27093 ts_manager.cc:194] Registered new tserver with Master: d5f92c9c0da7491bb79c03d4bcd85760 (127.24.20.130:39445)
I20260620 20:13:40.614362 27093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:38973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:13:40.629351 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44999
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:44581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:40.734642 27423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:40.734867 27423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:40.734906 27423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:40.738750 27423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:40.738797 27423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:40.738850 27423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:13:40.743824 27423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:44581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:40.744782 27423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:40.745718 27423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:13:40.754954 27423 server_base.cc:1034] running on GCE node
W20260620 20:13:40.754935 27429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:40.754935 27428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:40.755095 27431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:40.755518 27423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:40.756032 27423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:40.757191 27423 hybrid_clock.cc:648] HybridClock initialized: now 1781986420757163 us; error 39 us; skew 500 ppm
Jun 20 20:13:40 dist-test-slave-pflr krb5kdc[27048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986420, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:40.759709 27423 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:13:40.760520 27423 webserver.cc:460] Webserver started at http://127.24.20.131:42449/ using document root <none> and password file <none>
I20260620 20:13:40.761078 27423 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:40.761138 27423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:40.761293 27423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:40.763145 27423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "0e27c00988934a6183ab0cff8b9de968"
format_stamp: "Formatted at 2026-06-20 20:13:40 on dist-test-slave-pflr"
server_key: "2620000017e12d2ef12beee146d3336b"
server_key_iv: "e7ba1ca1e9807449336275fd003ec9df"
server_key_version: "encryptionkey@0"
I20260620 20:13:40.763634 27423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "0e27c00988934a6183ab0cff8b9de968"
format_stamp: "Formatted at 2026-06-20 20:13:40 on dist-test-slave-pflr"
server_key: "2620000017e12d2ef12beee146d3336b"
server_key_iv: "e7ba1ca1e9807449336275fd003ec9df"
server_key_version: "encryptionkey@0"
I20260620 20:13:40.767076 27423 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:13:40.769389 27438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:40.770511 27423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:40.770664 27423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "0e27c00988934a6183ab0cff8b9de968"
format_stamp: "Formatted at 2026-06-20 20:13:40 on dist-test-slave-pflr"
server_key: "2620000017e12d2ef12beee146d3336b"
server_key_iv: "e7ba1ca1e9807449336275fd003ec9df"
server_key_version: "encryptionkey@0"
I20260620 20:13:40.770776 27423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:40.783233 27423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:40.785926 27423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:40.786119 27423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:40.786654 27423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:40.787405 27423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:40.787472 27423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:40.787536 27423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:40.787588 27423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:40.796272 27423 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:36627
I20260620 20:13:40.796315 27551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:36627 every 8 connection(s)
I20260620 20:13:40.797224 27423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:13:40.804972 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 27423
I20260620 20:13:40.805088 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:13:40.805382 24658 external_mini_cluster.cc:1428] Setting key 0c0a2a2a3dcb0704db01c4cb6cf91941
Jun 20 20:13:40 dist-test-slave-pflr krb5kdc[27048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986420, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:40.810097 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:40.798910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:59995 (local address 127.24.20.190:44999)
0620 20:13:40.799039 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:40.799043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:40.799862 (+ 819us) server_negotiation.cc:407] Connection header received
0620 20:13:40.801238 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:40.801241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:40.801294 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:40.801402 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:40.802476 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.803005 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.804102 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.804319 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.804435 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:40.807094 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:40.807098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:40.807101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:40.807134 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:40.808405 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:40.808899 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:40.808901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:40.808902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:40.808970 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:40.809262 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:40.809264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:40.809265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:40.809415 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:40.809485 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:40.809901 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 20:13:40.809985 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:13:40.810801 27554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:40.799305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44999 (local address 127.24.20.131:59995)
0620 20:13:40.799706 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:40.799745 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:40.801033 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:40.801563 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:40.801569 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:40.801894 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:40.802329 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:40.802340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.803139 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.803142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:40.803982 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:40.803991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:40.804466 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:40.804470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:40.804684 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:40.805280 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:40.805295 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:40.806951 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:40.808501 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:40.808505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:40.808515 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:40.808816 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:40.809049 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:40.809051 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:40.809052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:40.809179 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:40.809502 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:40.809506 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:40.809707 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 20:13:40.809842 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 20:13:40.810058 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":107,"threads_started":1}
I20260620 20:13:40.811892 27552 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44999
I20260620 20:13:40.812101 27552 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:40.812525 27552 heartbeater.cc:511] Master 127.24.20.190:44999 requested a full tablet report, sending...
I20260620 20:13:40.813433 27093 ts_manager.cc:194] Registered new tserver with Master: 0e27c00988934a6183ab0cff8b9de968 (127.24.20.131:36627)
I20260620 20:13:40.813932 27093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:59995
I20260620 20:13:40.819007 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 20:13:40.831123 27132 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:13:40.829675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49630 (local address 127.24.20.190:44999)
0620 20:13:40.829828 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:40.829833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:40.829993 (+ 160us) server_negotiation.cc:407] Connection header received
0620 20:13:40.830165 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:40.830173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:40.830213 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:40.830297 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:40.831017 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49630: BlockingRecv error: recv got EOF from 127.0.0.1:49630 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:13:40.832170 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 27155
I20260620 20:13:40.837282 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 27289
I20260620 20:13:40.842341 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 27423
I20260620 20:13:40.847219 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 27061
2026-06-20T20:13:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2093 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:13:40 dist-test-slave-pflr krb5kdc[27564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:13:40 dist-test-slave-pflr krb5kdc[27564](info): set up 2 sockets
Jun 20 20:13:40 dist-test-slave-pflr krb5kdc[27564](info): commencing operation
krb5kdc: starting...
W20260620 20:13:41.880986 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:13:41 dist-test-slave-pflr krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:13:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:13:41.997802 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:43021
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:34857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:43021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:42.103292 27577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:42.103549 27577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:42.103617 27577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:42.107771 27577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:42.107868 27577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:42.107908 27577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:42.107961 27577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:42.107999 27577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:42.113329 27577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:34857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:43021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:43021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:42.114284 27577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:42.115229 27577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:42.123304 27585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:42.123298 27582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:42.123662 27577 server_base.cc:1034] running on GCE node
W20260620 20:13:42.123298 27583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:42.124204 27577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:42.125288 27577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:42.126470 27577 hybrid_clock.cc:648] HybridClock initialized: now 1781986422126458 us; error 26 us; skew 500 ppm
Jun 20 20:13:42 dist-test-slave-pflr krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986422, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:42.129122 27577 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:13:42.129947 27577 webserver.cc:460] Webserver started at http://127.24.20.190:46103/ using document root <none> and password file <none>
I20260620 20:13:42.130457 27577 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:42.130539 27577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:42.130759 27577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:42.132705 27577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "3b134c6f53cb4596b316572d36a833e8"
format_stamp: "Formatted at 2026-06-20 20:13:42 on dist-test-slave-pflr"
server_key: "f161017c20e2f1da96e8af2b4464368b"
server_key_iv: "9840b61986be68d1399bd980539ac2c7"
server_key_version: "encryptionkey@0"
I20260620 20:13:42.133232 27577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "3b134c6f53cb4596b316572d36a833e8"
format_stamp: "Formatted at 2026-06-20 20:13:42 on dist-test-slave-pflr"
server_key: "f161017c20e2f1da96e8af2b4464368b"
server_key_iv: "9840b61986be68d1399bd980539ac2c7"
server_key_version: "encryptionkey@0"
I20260620 20:13:42.136519 27577 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:13:42.138725 27592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:42.139835 27577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:42.139961 27577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "3b134c6f53cb4596b316572d36a833e8"
format_stamp: "Formatted at 2026-06-20 20:13:42 on dist-test-slave-pflr"
server_key: "f161017c20e2f1da96e8af2b4464368b"
server_key_iv: "9840b61986be68d1399bd980539ac2c7"
server_key_version: "encryptionkey@0"
I20260620 20:13:42.140057 27577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:42.159830 27577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:42.162889 27577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:42.163081 27577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:42.169812 27577 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:43021
I20260620 20:13:42.170084 27644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:43021 every 8 connection(s)
I20260620 20:13:42.170737 27577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:42.173209 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 27577
I20260620 20:13:42.173336 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:42.173619 27645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:42.173646 24658 external_mini_cluster.cc:1428] Setting key db4b2b560ac8dbf0bcc285016e4e1ca1
I20260620 20:13:42.180859 27645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8: Bootstrap starting.
Jun 20 20:13:42 dist-test-slave-pflr krb5kdc[27564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:42.183269 27645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:42.184177 27645 log.cc:824] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:42.186442 27645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8: No bootstrap required, opened a new log
I20260620 20:13:42.188748 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:42.174935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.24.20.190:43021)
0620 20:13:42.175330 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:42.175348 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:42.175392 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:13:42.176264 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:42.176289 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:42.176740 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:42.177122 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:42.177797 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:42.179247 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:42.180133 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:42.180824 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:42.180990 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:42.183640 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:42.183646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:42.183662 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:42.183693 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:42.185607 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:42.186022 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:42.186028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:42.186033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:42.186183 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:42.186577 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:42.186580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:42.186582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:42.187121 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:42.187301 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:42.187736 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 20:13:42.188004 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":103,"threads_started":1}
I20260620 20:13:42.190258 27645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b134c6f53cb4596b316572d36a833e8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 43021 } }
I20260620 20:13:42.190546 27645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:42.190641 27645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b134c6f53cb4596b316572d36a833e8, State: Initialized, Role: FOLLOWER
I20260620 20:13:42.191083 27645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b134c6f53cb4596b316572d36a833e8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 43021 } }
I20260620 20:13:42.191238 27645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:42.191326 27645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:42.191425 27645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:42.191447 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:13:42.192289 27645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b134c6f53cb4596b316572d36a833e8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 43021 } }
I20260620 20:13:42.192660 27645 leader_election.cc:302] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b134c6f53cb4596b316572d36a833e8; no voters:
I20260620 20:13:42.192965 27645 leader_election.cc:288] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:42.193112 27650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:42.193409 27650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [term 1 LEADER]: Becoming Leader. State: Replica: 3b134c6f53cb4596b316572d36a833e8, State: Running, Role: LEADER
I20260620 20:13:42.193730 27650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b134c6f53cb4596b316572d36a833e8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 43021 } }
I20260620 20:13:42.194314 27645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:42.195356 27653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b134c6f53cb4596b316572d36a833e8. Latest consensus state: current_term: 1 leader_uuid: "3b134c6f53cb4596b316572d36a833e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b134c6f53cb4596b316572d36a833e8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 43021 } } }
I20260620 20:13:42.195356 27652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b134c6f53cb4596b316572d36a833e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b134c6f53cb4596b316572d36a833e8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 43021 } } }
I20260620 20:13:42.195494 27652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:42.195494 27653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8 [sys.catalog]: This master's current role is: LEADER
W20260620 20:13:42.199342 27666 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:13:42.199417 27666 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:13:42.199473 27660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:42.200325 27660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:42.205468 27660 catalog_manager.cc:1347] Generated new cluster ID: 7709a2c5e5e741da9f4f00e828249790
I20260620 20:13:42.205523 27660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:42.211009 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:42.206266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.24.20.190:43021)
0620 20:13:42.206392 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:42.206397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:42.206653 (+ 256us) server_negotiation.cc:407] Connection header received
0620 20:13:42.206812 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:42.206815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:42.206865 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:42.206950 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:42.207458 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:42.208005 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:42.208923 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:42.209176 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:42.209289 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:42.210119 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:42.210122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:42.210125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:42.210516 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:42.210543 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:42.210558 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:42.210661 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:42.210791 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 20:13:42.210849 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":42}
I20260620 20:13:42.227300 27660 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:42.228232 27660 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:42.237391 27660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b134c6f53cb4596b316572d36a833e8: Generated new TSK 0
I20260620 20:13:42.238265 27660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:13:42.252249 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 27577
2026-06-20T20:13:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1402 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:13:42 dist-test-slave-pflr krb5kdc[27678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:13:42 dist-test-slave-pflr krb5kdc[27678](info): set up 2 sockets
Jun 20 20:13:42 dist-test-slave-pflr krb5kdc[27678](info): commencing operation
krb5kdc: starting...
W20260620 20:13:43.317307 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:13:43 dist-test-slave-pflr krb5kdc[27678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:13:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:13:43.437407 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:45611
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:45611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:43.543627 27691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:43.543896 27691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:43.543936 27691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:43.547917 27691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:43.547971 27691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:43.547991 27691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:43.548041 27691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:43.548076 27691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:43.553406 27691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:45611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:45611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:43.554390 27691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:43.555368 27691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:43.563405 27696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:43.563470 27691 server_base.cc:1034] running on GCE node
W20260620 20:13:43.563421 27697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:43.563424 27699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:43.564169 27691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:43.565246 27691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:43.566427 27691 hybrid_clock.cc:648] HybridClock initialized: now 1781986423566399 us; error 41 us; skew 500 ppm
Jun 20 20:13:43 dist-test-slave-pflr krb5kdc[27678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986423, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:43.569000 27691 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:13:43.569823 27691 webserver.cc:460] Webserver started at http://127.24.20.190:41855/ using document root <none> and password file <none>
I20260620 20:13:43.570322 27691 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:43.570375 27691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:43.570525 27691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:43.572417 27691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "a452113feee843838bc538c35fd673f8"
format_stamp: "Formatted at 2026-06-20 20:13:43 on dist-test-slave-pflr"
server_key: "7865a8c2199e06fedcaaa14cf2d66839"
server_key_iv: "b4a1219df0c5e96da64ec648dab14ebd"
server_key_version: "encryptionkey@0"
I20260620 20:13:43.572965 27691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "a452113feee843838bc538c35fd673f8"
format_stamp: "Formatted at 2026-06-20 20:13:43 on dist-test-slave-pflr"
server_key: "7865a8c2199e06fedcaaa14cf2d66839"
server_key_iv: "b4a1219df0c5e96da64ec648dab14ebd"
server_key_version: "encryptionkey@0"
I20260620 20:13:43.576280 27691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:13:43.578483 27706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:43.579612 27691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:43.579707 27691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "a452113feee843838bc538c35fd673f8"
format_stamp: "Formatted at 2026-06-20 20:13:43 on dist-test-slave-pflr"
server_key: "7865a8c2199e06fedcaaa14cf2d66839"
server_key_iv: "b4a1219df0c5e96da64ec648dab14ebd"
server_key_version: "encryptionkey@0"
I20260620 20:13:43.579805 27691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:43.595295 27691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:43.598001 27691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:43.598157 27691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:43.604835 27691 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:45611
I20260620 20:13:43.605137 27758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:45611 every 8 connection(s)
I20260620 20:13:43.605749 27691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:43.608310 27759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:43.613119 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 27691
I20260620 20:13:43.613255 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:43.613538 24658 external_mini_cluster.cc:1428] Setting key 524f82e833b42cd4f6808b66d8fc4213
I20260620 20:13:43.614499 27759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8: Bootstrap starting.
I20260620 20:13:43.616801 27759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:43.617663 27759 log.cc:824] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:43.619706 27759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8: No bootstrap required, opened a new log
Jun 20 20:13:43 dist-test-slave-pflr krb5kdc[27678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:43.624028 27759 raft_consensus.cc:348] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a452113feee843838bc538c35fd673f8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 45611 } }
I20260620 20:13:43.624265 27759 raft_consensus.cc:374] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:43.624346 27759 raft_consensus.cc:729] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a452113feee843838bc538c35fd673f8, State: Initialized, Role: FOLLOWER
I20260620 20:13:43.624830 27759 consensus_queue.cc:260] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a452113feee843838bc538c35fd673f8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 45611 } }
I20260620 20:13:43.624976 27759 raft_consensus.cc:388] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:43.625063 27759 raft_consensus.cc:482] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:43.625159 27759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:43.625970 27759 raft_consensus.cc:504] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a452113feee843838bc538c35fd673f8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 45611 } }
I20260620 20:13:43.626322 27759 leader_election.cc:302] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a452113feee843838bc538c35fd673f8; no voters:
I20260620 20:13:43.626670 27759 leader_election.cc:288] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:43.626879 27764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:43.627111 27764 raft_consensus.cc:686] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [term 1 LEADER]: Becoming Leader. State: Replica: a452113feee843838bc538c35fd673f8, State: Running, Role: LEADER
I20260620 20:13:43.627403 27764 consensus_queue.cc:237] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a452113feee843838bc538c35fd673f8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 45611 } }
I20260620 20:13:43.627820 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:43.614934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.24.20.190:45611)
0620 20:13:43.615306 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:43.615323 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:43.615369 (+ 46us) server_negotiation.cc:407] Connection header received
0620 20:13:43.615984 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:43.616001 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:43.616308 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:43.616566 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:43.617499 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:43.618538 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:43.619658 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:43.620138 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:43.620253 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:43.622945 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:43.622950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:43.622961 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:43.622985 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:43.624802 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:43.625248 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:43.625254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:43.625261 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:43.625376 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:43.625675 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:43.625679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:43.625680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:43.626044 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:43.626243 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:43.626526 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 20:13:43.626850 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":247,"thread_start_us":135,"threads_started":1}
I20260620 20:13:43.627867 27759 sys_catalog.cc:564] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:43.628973 27765 sys_catalog.cc:455] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a452113feee843838bc538c35fd673f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a452113feee843838bc538c35fd673f8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 45611 } } }
I20260620 20:13:43.629084 27765 sys_catalog.cc:458] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:43.629280 27765 sys_catalog.cc:455] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a452113feee843838bc538c35fd673f8. Latest consensus state: current_term: 1 leader_uuid: "a452113feee843838bc538c35fd673f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a452113feee843838bc538c35fd673f8" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 45611 } } }
I20260620 20:13:43.629345 27765 sys_catalog.cc:458] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:43.630267 27773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:43.633315 27773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:43.638465 27773 catalog_manager.cc:1347] Generated new cluster ID: f7eba05981ba4771997c33c3296df2f3
I20260620 20:13:43.638526 27773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:43.651484 27773 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:43.652654 27773 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:43.661209 27773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a452113feee843838bc538c35fd673f8: Generated new TSK 0
I20260620 20:13:43.661949 27773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:13:43.687952 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:45611
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:43.794766 27785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:43.795009 27785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:43.795049 27785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:43.799022 27785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:43.799078 27785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:43.799178 27785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:13:43.804317 27785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:45611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:43.805321 27785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:43.806278 27785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:43.815431 27790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:43.815454 27791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:43.815439 27793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:43.815770 27785 server_base.cc:1034] running on GCE node
I20260620 20:13:43.816080 27785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:43.816704 27785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:43.817862 27785 hybrid_clock.cc:648] HybridClock initialized: now 1781986423817849 us; error 23 us; skew 500 ppm
Jun 20 20:13:43 dist-test-slave-pflr krb5kdc[27678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986423, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:43.825915 27785 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:13:43.826764 27785 webserver.cc:460] Webserver started at http://127.24.20.129:42013/ using document root <none> and password file <none>
I20260620 20:13:43.827343 27785 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:43.827417 27785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:43.827651 27785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:43.829820 27785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "b5d9aaebf33447e68e4bfefa2524460a"
format_stamp: "Formatted at 2026-06-20 20:13:43 on dist-test-slave-pflr"
server_key: "9f1395db9adc8d106b7a2c4a225b07bb"
server_key_iv: "07989543da7c41ffa5eb5e16200fa258"
server_key_version: "encryptionkey@0"
I20260620 20:13:43.830370 27785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "b5d9aaebf33447e68e4bfefa2524460a"
format_stamp: "Formatted at 2026-06-20 20:13:43 on dist-test-slave-pflr"
server_key: "9f1395db9adc8d106b7a2c4a225b07bb"
server_key_iv: "07989543da7c41ffa5eb5e16200fa258"
server_key_version: "encryptionkey@0"
I20260620 20:13:43.833882 27785 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:13:43.836059 27800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:43.837301 27785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:43.837436 27785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "b5d9aaebf33447e68e4bfefa2524460a"
format_stamp: "Formatted at 2026-06-20 20:13:43 on dist-test-slave-pflr"
server_key: "9f1395db9adc8d106b7a2c4a225b07bb"
server_key_iv: "07989543da7c41ffa5eb5e16200fa258"
server_key_version: "encryptionkey@0"
I20260620 20:13:43.837549 27785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:43.855129 27785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:43.857864 27785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:43.858062 27785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:43.858605 27785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:43.859380 27785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:43.859447 27785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:43.859512 27785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:43.859566 27785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:43.867897 27785 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:33697
I20260620 20:13:43.867947 27913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:33697 every 8 connection(s)
I20260620 20:13:43.868831 27785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:43.873703 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 27785
I20260620 20:13:43.873829 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:13:43.874109 24658 external_mini_cluster.cc:1428] Setting key b539bff1b0f6a73a4150066008712d91
Jun 20 20:13:43 dist-test-slave-pflr krb5kdc[27678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986423, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:43.883343 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:43.870517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:39211 (local address 127.24.20.190:45611)
0620 20:13:43.870654 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:43.870657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:43.871439 (+ 782us) server_negotiation.cc:407] Connection header received
0620 20:13:43.872791 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:43.872795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:43.872853 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:43.872957 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:43.874206 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:43.874867 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:43.876458 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:43.876747 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:43.876827 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:43.879320 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:43.879324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:43.879327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:43.879358 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:43.881113 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:43.881626 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:43.881630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:43.881632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:43.881727 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:43.882037 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:43.882042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:43.882044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:43.882294 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:43.882406 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:43.882993 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 20:13:43.883121 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:13:43.884110 27916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:43.870862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:45611 (local address 127.24.20.129:39211)
0620 20:13:43.871306 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:43.871345 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:43.872538 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:43.873122 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:43.873130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:43.873493 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:43.874064 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:43.874078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:43.874993 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:43.874996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:43.876321 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:43.876336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:43.876889 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:43.876896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:43.877135 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:43.877793 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:43.877808 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:43.879149 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:43.881279 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:43.881283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:43.881293 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:43.881527 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:43.881821 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:43.881823 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:43.881824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:43.881942 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:43.882400 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:43.882405 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:43.882748 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 20:13:43.882926 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 20:13:43.883187 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20260620 20:13:43.885332 27914 heartbeater.cc:347] Connected to a master server at 127.24.20.190:45611
I20260620 20:13:43.885560 27914 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:43.886020 27914 heartbeater.cc:511] Master 127.24.20.190:45611 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
I20260620 20:13:43.887603 27723 ts_manager.cc:194] Registered new tserver with Master: b5d9aaebf33447e68e4bfefa2524460a (127.24.20.129:33697)
I20260620 20:13:43.888814 27723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:39211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:13:43.899672 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:45611
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:44.003611 27919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:44.003880 27919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:44.003955 27919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:44.007890 27919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:44.007970 27919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:44.008057 27919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:13:44.013038 27919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:45611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:44.013959 27919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:44.014887 27919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:44.023778 27927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:44.023926 27925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:44.023759 27924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:44.023838 27919 server_base.cc:1034] running on GCE node
I20260620 20:13:44.024371 27919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:44.024986 27919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:44.026160 27919 hybrid_clock.cc:648] HybridClock initialized: now 1781986424026150 us; error 29 us; skew 500 ppm
Jun 20 20:13:44 dist-test-slave-pflr krb5kdc[27678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986424, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:44.028798 27919 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:13:44.029639 27919 webserver.cc:460] Webserver started at http://127.24.20.130:41989/ using document root <none> and password file <none>
I20260620 20:13:44.030210 27919 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:44.030292 27919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:44.030488 27919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:44.032351 27919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "c79345c4169a42f1862e44dea91d8984"
format_stamp: "Formatted at 2026-06-20 20:13:44 on dist-test-slave-pflr"
server_key: "5a32b926a714534526e0ab78eaec4675"
server_key_iv: "51459ab3a7c18ec264dd7c3b78b193d3"
server_key_version: "encryptionkey@0"
I20260620 20:13:44.032905 27919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "c79345c4169a42f1862e44dea91d8984"
format_stamp: "Formatted at 2026-06-20 20:13:44 on dist-test-slave-pflr"
server_key: "5a32b926a714534526e0ab78eaec4675"
server_key_iv: "51459ab3a7c18ec264dd7c3b78b193d3"
server_key_version: "encryptionkey@0"
I20260620 20:13:44.036203 27919 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:13:44.038451 27934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:44.039577 27919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:44.039708 27919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "c79345c4169a42f1862e44dea91d8984"
format_stamp: "Formatted at 2026-06-20 20:13:44 on dist-test-slave-pflr"
server_key: "5a32b926a714534526e0ab78eaec4675"
server_key_iv: "51459ab3a7c18ec264dd7c3b78b193d3"
server_key_version: "encryptionkey@0"
I20260620 20:13:44.039811 27919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:44.057924 27919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:44.060503 27919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:44.060715 27919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:44.061234 27919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:44.061988 27919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:44.062055 27919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:44.062121 27919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:44.062156 27919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:44.070384 27919 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:40741
I20260620 20:13:44.070416 28047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:40741 every 8 connection(s)
I20260620 20:13:44.071177 27919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:44.074745 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 27919
I20260620 20:13:44.074838 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:13:44.075140 24658 external_mini_cluster.cc:1428] Setting key 7018930c8d3e796f0cca8152c0c66c5f
Jun 20 20:13:44 dist-test-slave-pflr krb5kdc[27678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986424, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
I20260620 20:13:44.085161 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:44.072754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:43247 (local address 127.24.20.190:45611)
0620 20:13:44.072880 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:44.072884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:44.073660 (+ 776us) server_negotiation.cc:407] Connection header received
0620 20:13:44.075089 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:44.075093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:44.075166 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:44.075271 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:44.076576 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:44.077125 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:44.078301 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:44.078621 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:44.078731 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:44.081285 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:44.081290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:44.081296 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:44.081327 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:44.083019 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:44.083546 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:44.083551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:44.083553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:44.083648 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:44.083990 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:44.083993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:44.083995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:44.084230 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:44.084342 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:44.084875 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 20:13:44.084994 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:13:44.085950 28050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:44.073113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:45611 (local address 127.24.20.130:43247)
0620 20:13:44.073514 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:44.073553 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:44.074846 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:44.075477 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:44.075483 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:44.075858 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:44.076437 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:44.076452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:44.077253 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:44.077257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:44.078184 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:44.078193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:44.078764 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:44.078768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:44.078930 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:44.079467 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:44.079492 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:44.081120 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:44.083179 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:44.083183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:44.083193 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:44.083425 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:44.083741 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:44.083746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:44.083748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:44.083877 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:44.084349 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:44.084357 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:44.084580 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 20:13:44.084820 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 20:13:44.085114 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":107,"threads_started":1}
I20260620 20:13:44.087018 28048 heartbeater.cc:347] Connected to a master server at 127.24.20.190:45611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
I20260620 20:13:44.087247 28048 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:44.087775 28048 heartbeater.cc:511] Master 127.24.20.190:45611 requested a full tablet report, sending...
I20260620 20:13:44.088883 27723 ts_manager.cc:194] Registered new tserver with Master: c79345c4169a42f1862e44dea91d8984 (127.24.20.130:40741)
I20260620 20:13:44.089692 27723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:43247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:13:44.101292 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:45611
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:38091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:44.207744 28053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:44.207978 28053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:44.208019 28053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:44.211987 28053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:44.212065 28053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:44.212169 28053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:13:44.217239 28053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:45611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:44.218123 28053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:44.219010 28053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:44.228336 28058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:44.228396 28059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:44.228348 28061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:44.228457 28053 server_base.cc:1034] running on GCE node
I20260620 20:13:44.228931 28053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:44.229573 28053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:44.230728 28053 hybrid_clock.cc:648] HybridClock initialized: now 1781986424230710 us; error 29 us; skew 500 ppm
Jun 20 20:13:44 dist-test-slave-pflr krb5kdc[27678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986424, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:44.233275 28053 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:13:44.234012 28053 webserver.cc:460] Webserver started at http://127.24.20.131:44453/ using document root <none> and password file <none>
I20260620 20:13:44.234493 28053 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:44.234547 28053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:44.234704 28053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:44.236712 28053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "0c3bcddce6bc4860bdd375d559d9eb20"
format_stamp: "Formatted at 2026-06-20 20:13:44 on dist-test-slave-pflr"
server_key: "c5cbcf9a3d900e15b1c48b0423d19d23"
server_key_iv: "39f0eccffc0c3a9511322bae4095e3ad"
server_key_version: "encryptionkey@0"
I20260620 20:13:44.237203 28053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "0c3bcddce6bc4860bdd375d559d9eb20"
format_stamp: "Formatted at 2026-06-20 20:13:44 on dist-test-slave-pflr"
server_key: "c5cbcf9a3d900e15b1c48b0423d19d23"
server_key_iv: "39f0eccffc0c3a9511322bae4095e3ad"
server_key_version: "encryptionkey@0"
I20260620 20:13:44.240525 28053 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:13:44.242888 28068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:44.244095 28053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:44.244215 28053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "0c3bcddce6bc4860bdd375d559d9eb20"
format_stamp: "Formatted at 2026-06-20 20:13:44 on dist-test-slave-pflr"
server_key: "c5cbcf9a3d900e15b1c48b0423d19d23"
server_key_iv: "39f0eccffc0c3a9511322bae4095e3ad"
server_key_version: "encryptionkey@0"
I20260620 20:13:44.244311 28053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:44.261457 28053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:44.264683 28053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:44.264870 28053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:44.265413 28053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:44.266161 28053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:44.266206 28053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:44.266271 28053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:44.266304 28053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:44.274360 28053 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:36381
I20260620 20:13:44.274428 28181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:36381 every 8 connection(s)
I20260620 20:13:44.275202 28053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:13:44.276671 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 28053
I20260620 20:13:44.276774 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:13:44.277052 24658 external_mini_cluster.cc:1428] Setting key efe1e5b017ba243f9beea12e09fbb709
Jun 20 20:13:44 dist-test-slave-pflr krb5kdc[27678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986424, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:44.288682 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:44.276823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:36785 (local address 127.24.20.190:45611)
0620 20:13:44.276955 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:44.276960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:44.277853 (+ 893us) server_negotiation.cc:407] Connection header received
0620 20:13:44.279551 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:44.279553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:44.279598 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:44.279669 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:44.280724 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:44.281224 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:44.282249 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:44.282461 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:44.282532 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:44.285329 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:44.285332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:44.285335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:44.285356 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:44.286878 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:44.287343 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:44.287345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:44.287346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:44.287425 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:44.287763 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:44.287765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:44.287766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:44.287918 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:44.287988 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:44.288425 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 20:13:44.288513 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:13:44.289417 28184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:44.277253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:45611 (local address 127.24.20.131:36785)
0620 20:13:44.277712 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:44.277765 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:44.279376 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:44.279817 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:44.279823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:44.280161 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:44.280611 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:44.280624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:44.281321 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:44.281323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:44.282152 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:44.282159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:44.282577 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:44.282580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:44.282877 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:44.283530 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:44.283545 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:44.285172 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:44.286987 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:44.286992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:44.287002 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:44.287248 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:44.287537 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:44.287539 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:44.287541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:44.287680 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:44.288020 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:44.288025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:44.288226 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 20:13:44.288373 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 20:13:44.288609 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
I20260620 20:13:44.290485 28182 heartbeater.cc:347] Connected to a master server at 127.24.20.190:45611
I20260620 20:13:44.290697 28182 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:44.291141 28182 heartbeater.cc:511] Master 127.24.20.190:45611 requested a full tablet report, sending...
I20260620 20:13:44.292053 27722 ts_manager.cc:194] Registered new tserver with Master: 0c3bcddce6bc4860bdd375d559d9eb20 (127.24.20.131:36381)
I20260620 20:13:44.292567 27722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:36785
I20260620 20:13:44.302671 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:13:44 dist-test-slave-pflr krb5kdc[27678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986424, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 20:13:44 dist-test-slave-pflr krb5kdc[27678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986424, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:44.326166 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:44.318341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.24.20.190:45611)
0620 20:13:44.318507 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:44.318510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:44.318624 (+ 114us) server_negotiation.cc:407] Connection header received
0620 20:13:44.318735 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:44.318738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:44.318778 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:44.318843 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:44.319455 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:44.319955 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:44.320773 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:44.320970 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:44.321046 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:44.323303 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:44.323306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:44.323308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:44.323329 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:44.324775 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:44.325138 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:44.325140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:44.325141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:44.325207 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:44.325418 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:44.325420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:44.325421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:44.325607 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:44.325714 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:44.325879 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 20:13:44.325959 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 20:13:44.326481 27722 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42434
I20260620 20:13:44.327242 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 27785
I20260620 20:13:44.332430 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 27919
I20260620 20:13:44.337379 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 28053
I20260620 20:13:44.342048 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 27691
2026-06-20T20:13:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2091 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:13:44 dist-test-slave-pflr krb5kdc[28194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:13:44 dist-test-slave-pflr krb5kdc[28194](info): set up 2 sockets
Jun 20 20:13:44 dist-test-slave-pflr krb5kdc[28194](info): commencing operation
krb5kdc: starting...
W20260620 20:13:45.401333 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:13:45 dist-test-slave-pflr krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:13:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:13:45.519042 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:37125
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:45123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:37125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:45.626160 28207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:45.626428 28207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:45.626505 28207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:45.630470 28207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:45.630548 28207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:45.630584 28207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:45.630646 28207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:45.630682 28207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:45.635800 28207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:45123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:37125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:37125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:45.636832 28207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:45.637773 28207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:45.645908 28213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:45.645915 28215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:45.645915 28212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:45.646106 28207 server_base.cc:1034] running on GCE node
I20260620 20:13:45.646711 28207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:45.647770 28207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:45.648931 28207 hybrid_clock.cc:648] HybridClock initialized: now 1781986425648902 us; error 37 us; skew 500 ppm
Jun 20 20:13:45 dist-test-slave-pflr krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986425, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:45.651603 28207 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:13:45.652438 28207 webserver.cc:460] Webserver started at http://127.24.20.190:46719/ using document root <none> and password file <none>
I20260620 20:13:45.653050 28207 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:45.653138 28207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:45.653363 28207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:45.655350 28207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "e717192d707d4504a4cd9e485767b281"
format_stamp: "Formatted at 2026-06-20 20:13:45 on dist-test-slave-pflr"
server_key: "5c695a18ab9a22b21880be6cbadfd8ae"
server_key_iv: "012875b7ab37190ea3089ac367f3f276"
server_key_version: "encryptionkey@0"
I20260620 20:13:45.655902 28207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "e717192d707d4504a4cd9e485767b281"
format_stamp: "Formatted at 2026-06-20 20:13:45 on dist-test-slave-pflr"
server_key: "5c695a18ab9a22b21880be6cbadfd8ae"
server_key_iv: "012875b7ab37190ea3089ac367f3f276"
server_key_version: "encryptionkey@0"
I20260620 20:13:45.659543 28207 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:13:45.661916 28222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:45.663112 28207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:45.663264 28207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "e717192d707d4504a4cd9e485767b281"
format_stamp: "Formatted at 2026-06-20 20:13:45 on dist-test-slave-pflr"
server_key: "5c695a18ab9a22b21880be6cbadfd8ae"
server_key_iv: "012875b7ab37190ea3089ac367f3f276"
server_key_version: "encryptionkey@0"
I20260620 20:13:45.663373 28207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:45.681420 28207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:45.684104 28207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:45.684302 28207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:45.691268 28207 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:37125
I20260620 20:13:45.691300 28274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:37125 every 8 connection(s)
I20260620 20:13:45.692178 28207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:45.694841 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 28207
I20260620 20:13:45.694868 28275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:45.694978 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:45.695297 24658 external_mini_cluster.cc:1428] Setting key 7643703281b0089832aa944690f5f284
I20260620 20:13:45.701086 28275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281: Bootstrap starting.
I20260620 20:13:45.703372 28275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281: Neither blocks nor log segments found. Creating new log.
Jun 20 20:13:45 dist-test-slave-pflr krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:45.704259 28275 log.cc:824] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:45.706101 28275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281: No bootstrap required, opened a new log
I20260620 20:13:45.709504 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:45.696714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43604 (local address 127.24.20.190:37125)
0620 20:13:45.697131 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:45.697145 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:45.697180 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:13:45.698048 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:45.698067 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:45.698391 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:45.698744 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:45.699577 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:45.700550 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:45.701503 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:45.701978 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:45.702166 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:45.704895 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:45.704901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:45.704918 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:45.704952 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:45.706705 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:45.707175 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:45.707179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:45.707183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:45.707286 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:45.707576 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:45.707578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:45.707579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:45.707944 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:45.708079 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:45.708437 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 20:13:45.708716 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260620 20:13:45.710216 28275 raft_consensus.cc:348] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e717192d707d4504a4cd9e485767b281" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37125 } }
I20260620 20:13:45.710469 28275 raft_consensus.cc:374] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:45.710549 28275 raft_consensus.cc:729] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e717192d707d4504a4cd9e485767b281, State: Initialized, Role: FOLLOWER
I20260620 20:13:45.710990 28275 consensus_queue.cc:260] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e717192d707d4504a4cd9e485767b281" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37125 } }
I20260620 20:13:45.711138 28275 raft_consensus.cc:388] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:45.711200 28275 raft_consensus.cc:482] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:45.711292 28275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:45.712211 28275 raft_consensus.cc:504] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e717192d707d4504a4cd9e485767b281" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37125 } }
I20260620 20:13:45.712491 28275 leader_election.cc:302] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e717192d707d4504a4cd9e485767b281; no voters:
I20260620 20:13:45.712813 28275 leader_election.cc:288] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:45.712893 28280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:45.713114 28280 raft_consensus.cc:686] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [term 1 LEADER]: Becoming Leader. State: Replica: e717192d707d4504a4cd9e485767b281, State: Running, Role: LEADER
I20260620 20:13:45.713409 28280 consensus_queue.cc:237] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e717192d707d4504a4cd9e485767b281" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37125 } }
I20260620 20:13:45.713979 28275 sys_catalog.cc:564] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:45.714855 28281 sys_catalog.cc:455] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e717192d707d4504a4cd9e485767b281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e717192d707d4504a4cd9e485767b281" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37125 } } }
I20260620 20:13:45.714990 28281 sys_catalog.cc:458] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:45.715359 28289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:45.715577 28282 sys_catalog.cc:455] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e717192d707d4504a4cd9e485767b281. Latest consensus state: current_term: 1 leader_uuid: "e717192d707d4504a4cd9e485767b281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e717192d707d4504a4cd9e485767b281" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37125 } } }
I20260620 20:13:45.715667 28282 sys_catalog.cc:458] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:45.718389 28289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:45.723379 28289 catalog_manager.cc:1347] Generated new cluster ID: 0b97976c5beb401c8d014f2742aaaa2f
I20260620 20:13:45.723440 28289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:45.731863 28289 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:45.732993 28289 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:45.739289 28289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e717192d707d4504a4cd9e485767b281: Generated new TSK 0
I20260620 20:13:45.739907 28289 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:13:45.771795 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:37125
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:45123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:45.879670 28301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:45.879941 28301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:45.880013 28301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:45.884161 28301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:45.884243 28301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:45.884333 28301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:13:45.889628 28301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:45123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:37125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:45.890597 28301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:45.891553 28301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:45.900722 28307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:45.900712 28306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:45.900755 28301 server_base.cc:1034] running on GCE node
W20260620 20:13:45.900712 28309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:45.901377 28301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:45.901897 28301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:45.903059 28301 hybrid_clock.cc:648] HybridClock initialized: now 1781986425903039 us; error 29 us; skew 500 ppm
Jun 20 20:13:45 dist-test-slave-pflr krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986425, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:45.905685 28301 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:13:45.906499 28301 webserver.cc:460] Webserver started at http://127.24.20.129:40221/ using document root <none> and password file <none>
I20260620 20:13:45.907028 28301 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:45.907085 28301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:45.907294 28301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:45.909267 28301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "0e4564c8a9ab4411b1229fdd741625c1"
format_stamp: "Formatted at 2026-06-20 20:13:45 on dist-test-slave-pflr"
server_key: "4d5be093712f008f3c30fba1e47a5aac"
server_key_iv: "28616fb377c9d6857533bdcb44d16798"
server_key_version: "encryptionkey@0"
I20260620 20:13:45.909806 28301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "0e4564c8a9ab4411b1229fdd741625c1"
format_stamp: "Formatted at 2026-06-20 20:13:45 on dist-test-slave-pflr"
server_key: "4d5be093712f008f3c30fba1e47a5aac"
server_key_iv: "28616fb377c9d6857533bdcb44d16798"
server_key_version: "encryptionkey@0"
I20260620 20:13:45.913303 28301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:13:45.915773 28316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:45.916915 28301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:45.917025 28301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "0e4564c8a9ab4411b1229fdd741625c1"
format_stamp: "Formatted at 2026-06-20 20:13:45 on dist-test-slave-pflr"
server_key: "4d5be093712f008f3c30fba1e47a5aac"
server_key_iv: "28616fb377c9d6857533bdcb44d16798"
server_key_version: "encryptionkey@0"
I20260620 20:13:45.917119 28301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:45.929958 28301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:45.932444 28301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:45.932685 28301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:45.933192 28301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:45.933914 28301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:45.933981 28301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:45.934046 28301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:45.934093 28301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:45.942397 28301 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:35001
I20260620 20:13:45.942436 28429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:35001 every 8 connection(s)
I20260620 20:13:45.943184 28301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:45.947829 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 28301
I20260620 20:13:45.947975 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:13:45.948251 24658 external_mini_cluster.cc:1428] Setting key 6771cab95b052aa5161ad18bce507086
Jun 20 20:13:45 dist-test-slave-pflr krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986425, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:45.957127 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:45.944807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:37189 (local address 127.24.20.190:37125)
0620 20:13:45.944931 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:45.944934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:45.945675 (+ 741us) server_negotiation.cc:407] Connection header received
0620 20:13:45.947009 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:45.947012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:45.947068 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:45.947159 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:45.948292 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:45.949012 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:45.950620 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:45.950845 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:45.950924 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:45.953378 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:45.953383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:45.953389 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:45.953422 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:45.955081 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:45.955547 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:45.955553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:45.955556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:45.955650 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:45.955975 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:45.955978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:45.955980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:45.956228 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:45.956333 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:45.956807 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 20:13:45.956939 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:13:45.957839 28432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:45.945129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:37125 (local address 127.24.20.129:37189)
0620 20:13:45.945534 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:45.945571 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:45.946796 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:45.947322 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:45.947328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:45.947651 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:45.948159 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:45.948173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:45.949137 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:45.949140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:45.950492 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:45.950507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:45.950993 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:45.951000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:45.951222 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:45.951805 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:45.951821 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:45.953222 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:45.955199 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:45.955203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:45.955212 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:45.955443 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:45.955740 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:45.955742 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:45.955743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:45.955875 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:45.956330 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:45.956334 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:45.956523 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 20:13:45.956726 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 20:13:45.956943 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260620 20:13:45.958979 28430 heartbeater.cc:347] Connected to a master server at 127.24.20.190:37125
I20260620 20:13:45.959195 28430 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:45.959656 28430 heartbeater.cc:511] Master 127.24.20.190:37125 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
I20260620 20:13:45.961081 28239 ts_manager.cc:194] Registered new tserver with Master: 0e4564c8a9ab4411b1229fdd741625c1 (127.24.20.129:35001)
I20260620 20:13:45.962637 28239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:37189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:13:45.975657 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:37125
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:45123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:46.081758 28435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:46.082064 28435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:46.082141 28435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:46.086205 28435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:46.086261 28435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:46.086357 28435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:13:46.091697 28435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:45123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:37125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:46.092702 28435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:46.093685 28435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:46.102823 28441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:46.102855 28435 server_base.cc:1034] running on GCE node
W20260620 20:13:46.102830 28443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:46.102823 28440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:46.103438 28435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:46.103993 28435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:46.105144 28435 hybrid_clock.cc:648] HybridClock initialized: now 1781986426105130 us; error 25 us; skew 500 ppm
Jun 20 20:13:46 dist-test-slave-pflr krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986426, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:46.107615 28435 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:13:46.108317 28435 webserver.cc:460] Webserver started at http://127.24.20.130:34655/ using document root <none> and password file <none>
I20260620 20:13:46.108872 28435 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:46.108932 28435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:46.109081 28435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:46.111011 28435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "349736dd4b6d4ccd96db59af70d8dc7e"
format_stamp: "Formatted at 2026-06-20 20:13:46 on dist-test-slave-pflr"
server_key: "26eb35a16adb380efab015fb26a061c1"
server_key_iv: "8c71e80c8a2b6015d6967d3b202e819e"
server_key_version: "encryptionkey@0"
I20260620 20:13:46.111487 28435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "349736dd4b6d4ccd96db59af70d8dc7e"
format_stamp: "Formatted at 2026-06-20 20:13:46 on dist-test-slave-pflr"
server_key: "26eb35a16adb380efab015fb26a061c1"
server_key_iv: "8c71e80c8a2b6015d6967d3b202e819e"
server_key_version: "encryptionkey@0"
I20260620 20:13:46.114970 28435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:13:46.117205 28450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:46.118304 28435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:46.118412 28435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "349736dd4b6d4ccd96db59af70d8dc7e"
format_stamp: "Formatted at 2026-06-20 20:13:46 on dist-test-slave-pflr"
server_key: "26eb35a16adb380efab015fb26a061c1"
server_key_iv: "8c71e80c8a2b6015d6967d3b202e819e"
server_key_version: "encryptionkey@0"
I20260620 20:13:46.118505 28435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:46.131724 28435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:46.134759 28435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:46.134941 28435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:46.135460 28435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:46.136183 28435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:46.136231 28435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:46.136296 28435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:46.136332 28435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:46.144888 28435 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:40671
I20260620 20:13:46.144915 28563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:40671 every 8 connection(s)
I20260620 20:13:46.145687 28435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:46.151162 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 28435
I20260620 20:13:46.151288 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:13:46.151592 24658 external_mini_cluster.cc:1428] Setting key 0cc11f8b40f11224d09a3fd10c8a4beb
Jun 20 20:13:46 dist-test-slave-pflr krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986426, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:46.159271 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:46.147236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:45085 (local address 127.24.20.190:37125)
0620 20:13:46.147366 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:46.147370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:46.148091 (+ 721us) server_negotiation.cc:407] Connection header received
0620 20:13:46.149475 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:46.149478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:46.149534 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:46.149641 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:46.150685 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:46.151233 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:46.152710 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:46.153029 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:46.153112 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:46.155659 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:46.155662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:46.155663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:46.155684 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:46.157198 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:46.157768 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:46.157771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:46.157772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:46.157841 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:46.158231 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:46.158233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:46.158234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:46.158387 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:46.158456 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:46.159061 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 20:13:46.159157 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:13:46.160360 28566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:46.147544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:37125 (local address 127.24.20.130:45085)
0620 20:13:46.147953 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:46.147990 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:46.149281 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:46.149812 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:46.149818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:46.150121 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:46.150540 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:46.150552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:46.151396 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:46.151401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:46.152531 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:46.152540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:46.153187 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:46.153191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:46.153374 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:46.153923 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:46.153947 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:46.155492 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:46.157309 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:46.157316 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:46.157330 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:46.157666 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:46.157928 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:46.157933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:46.157938 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:46.158130 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:46.158486 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:46.158494 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:46.158806 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 20:13:46.159022 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 20:13:46.159361 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
I20260620 20:13:46.161545 28564 heartbeater.cc:347] Connected to a master server at 127.24.20.190:37125
I20260620 20:13:46.161759 28564 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:46.162202 28564 heartbeater.cc:511] Master 127.24.20.190:37125 requested a full tablet report, sending...
I20260620 20:13:46.163246 28239 ts_manager.cc:194] Registered new tserver with Master: 349736dd4b6d4ccd96db59af70d8dc7e (127.24.20.130:40671)
I20260620 20:13:46.163784 28239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:45085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:13:46.179049 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:37125
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:45123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:46.286844 28569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:46.287083 28569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:46.287124 28569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:46.291349 28569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:46.291399 28569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:46.291491 28569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:13:46.296790 28569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:45123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:37125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:46.297873 28569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:46.298871 28569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:46.308117 28577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:46.308136 28569 server_base.cc:1034] running on GCE node
W20260620 20:13:46.308046 28574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:46.308058 28575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:46.308719 28569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:46.309302 28569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:46.310492 28569 hybrid_clock.cc:648] HybridClock initialized: now 1781986426310471 us; error 31 us; skew 500 ppm
Jun 20 20:13:46 dist-test-slave-pflr krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986426, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:46.313177 28569 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:13:46.314047 28569 webserver.cc:460] Webserver started at http://127.24.20.131:39339/ using document root <none> and password file <none>
I20260620 20:13:46.314628 28569 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:46.314720 28569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:46.314944 28569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:46.317121 28569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "ccc4ecf160934e12af001681cd630c3c"
format_stamp: "Formatted at 2026-06-20 20:13:46 on dist-test-slave-pflr"
server_key: "3a6be6f01748054a27f6f9d0f82f759a"
server_key_iv: "d5f4de42dfa6e871b009df1d025eeaf0"
server_key_version: "encryptionkey@0"
I20260620 20:13:46.317668 28569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "ccc4ecf160934e12af001681cd630c3c"
format_stamp: "Formatted at 2026-06-20 20:13:46 on dist-test-slave-pflr"
server_key: "3a6be6f01748054a27f6f9d0f82f759a"
server_key_iv: "d5f4de42dfa6e871b009df1d025eeaf0"
server_key_version: "encryptionkey@0"
I20260620 20:13:46.321205 28569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:13:46.323578 28584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:46.324832 28569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 20:13:46.324954 28569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "ccc4ecf160934e12af001681cd630c3c"
format_stamp: "Formatted at 2026-06-20 20:13:46 on dist-test-slave-pflr"
server_key: "3a6be6f01748054a27f6f9d0f82f759a"
server_key_iv: "d5f4de42dfa6e871b009df1d025eeaf0"
server_key_version: "encryptionkey@0"
I20260620 20:13:46.325052 28569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:46.367902 28569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:46.370739 28569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:46.370946 28569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:46.371563 28569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:46.372354 28569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:46.372422 28569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:46.372494 28569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:46.372541 28569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:46.381346 28569 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:34613
I20260620 20:13:46.381373 28697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:34613 every 8 connection(s)
I20260620 20:13:46.382201 28569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:13:46.384876 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 28569
I20260620 20:13:46.384990 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:13:46.385308 24658 external_mini_cluster.cc:1428] Setting key 1041ccda3d622f600ddcd3fad2055fb0
Jun 20 20:13:46 dist-test-slave-pflr krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986426, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:46.395480 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:46.383916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:47033 (local address 127.24.20.190:37125)
0620 20:13:46.384085 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:46.384090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:46.384931 (+ 841us) server_negotiation.cc:407] Connection header received
0620 20:13:46.386360 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:46.386363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:46.386418 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:46.386504 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:46.387730 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:46.388368 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:46.389495 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:46.389757 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:46.389878 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:46.392194 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:46.392197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:46.392200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:46.392225 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:46.393577 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:46.394056 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:46.394061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:46.394063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:46.394152 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:46.394477 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:46.394480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:46.394482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:46.394657 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:46.394762 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:46.395206 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 20:13:46.395318 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:13:46.396204 28700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:46.384296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:37125 (local address 127.24.20.131:47033)
0620 20:13:46.384789 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:46.384828 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:46.386153 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:46.386692 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:46.386699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:46.387047 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:46.387586 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:46.387600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:46.388492 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:46.388496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:46.389372 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:46.389381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:46.389910 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:46.389914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:46.390080 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:46.390681 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:46.390696 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:46.392039 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:46.393698 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:46.393704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:46.393717 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:46.393958 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:46.394249 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:46.394252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:46.394253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:46.394380 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:46.394758 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:46.394764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:46.394986 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 20:13:46.395166 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 20:13:46.395455 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":110,"threads_started":1}
I20260620 20:13:46.397357 28698 heartbeater.cc:347] Connected to a master server at 127.24.20.190:37125
I20260620 20:13:46.397642 28698 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:46.398195 28698 heartbeater.cc:511] Master 127.24.20.190:37125 requested a full tablet report, sending...
I20260620 20:13:46.399224 28239 ts_manager.cc:194] Registered new tserver with Master: ccc4ecf160934e12af001681cd630c3c (127.24.20.131:34613)
I20260620 20:13:46.399763 28239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:47033
I20260620 20:13:46.410374 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:13:46 dist-test-slave-pflr krb5kdc[28194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:13:46 dist-test-slave-pflr krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.129@KRBTEST.COM
I20260620 20:13:46.435984 28705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:46.426605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.24.20.129:35001)
0620 20:13:46.426947 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:46.426961 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:46.427012 (+ 51us) server_negotiation.cc:407] Connection header received
0620 20:13:46.427059 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:46.427064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:46.427245 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:46.427354 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:46.428000 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:46.429058 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:46.430044 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:46.430405 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:46.430487 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:46.432954 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:46.432962 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:46.432974 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:46.432995 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:46.434422 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:46.434841 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:46.434846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:46.434848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:46.434930 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:46.435163 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:46.435165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:46.435166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:46.435373 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:46.435524 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:46.435727 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 20:13:46.435840 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":149,"threads_started":1}
Jun 20 20:13:46 dist-test-slave-pflr krb5kdc[28194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:46.446986 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:46.438755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.24.20.190:37125)
0620 20:13:46.438888 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:46.438895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:46.438933 (+ 38us) server_negotiation.cc:407] Connection header received
0620 20:13:46.439083 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:46.439085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:46.439138 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:46.439221 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:46.439786 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:46.440299 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:46.441124 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:46.441357 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:46.441465 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:46.444013 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:46.444019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:46.444026 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:46.444048 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:46.445358 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:46.445783 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:46.445786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:46.445788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:46.445863 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:46.446146 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:46.446149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:46.446151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:46.446351 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:46.446494 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:46.446715 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 20:13:46.446830 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 20:13:46.447351 28239 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49092
I20260620 20:13:46.447942 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 28301
I20260620 20:13:46.453358 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 28435
I20260620 20:13:46.458565 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 28569
I20260620 20:13:46.463613 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 28207
2026-06-20T20:13:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2121 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:13:46 dist-test-slave-pflr krb5kdc[28710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:13:46 dist-test-slave-pflr krb5kdc[28710](info): set up 2 sockets
Jun 20 20:13:46 dist-test-slave-pflr krb5kdc[28710](info): commencing operation
krb5kdc: starting...
W20260620 20:13:47.500944 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:13:47 dist-test-slave-pflr krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:13:47.616850 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:41663
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:41663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:47.723348 28723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:47.723584 28723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:47.723652 28723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:47.727528 28723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:47.727577 28723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:47.727596 28723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:47.727653 28723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:47.727686 28723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:47.732741 28723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:41663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:41663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:47.733685 28723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:47.734658 28723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:47.742504 28729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:47.742619 28723 server_base.cc:1034] running on GCE node
W20260620 20:13:47.742519 28731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:47.742889 28728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:47.743403 28723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:47.744433 28723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:47.745615 28723 hybrid_clock.cc:648] HybridClock initialized: now 1781986427745586 us; error 40 us; skew 500 ppm
Jun 20 20:13:47 dist-test-slave-pflr krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986427, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:47.748345 28723 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:13:47.749089 28723 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:47.749179 28723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:47.749394 28723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:47.751353 28723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "1642648f6efa490385b379084cf83cc5"
format_stamp: "Formatted at 2026-06-20 20:13:47 on dist-test-slave-pflr"
server_key: "f3e7250c62380309ff7f0d21aee50b0c"
server_key_iv: "26c3df9367c65d3781f1e25806c42e62"
server_key_version: "encryptionkey@0"
I20260620 20:13:47.751912 28723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "1642648f6efa490385b379084cf83cc5"
format_stamp: "Formatted at 2026-06-20 20:13:47 on dist-test-slave-pflr"
server_key: "f3e7250c62380309ff7f0d21aee50b0c"
server_key_iv: "26c3df9367c65d3781f1e25806c42e62"
server_key_version: "encryptionkey@0"
I20260620 20:13:47.755388 28723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:13:47.757682 28737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:47.758896 28723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:47.759126 28723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "1642648f6efa490385b379084cf83cc5"
format_stamp: "Formatted at 2026-06-20 20:13:47 on dist-test-slave-pflr"
server_key: "f3e7250c62380309ff7f0d21aee50b0c"
server_key_iv: "26c3df9367c65d3781f1e25806c42e62"
server_key_version: "encryptionkey@0"
I20260620 20:13:47.759234 28723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:47.776787 28723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:47.779938 28723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:47.780145 28723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:47.786746 28723 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:41663
I20260620 20:13:47.787012 28789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:41663 every 8 connection(s)
I20260620 20:13:47.787398 28723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:47.790030 28790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:47.792158 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 28723
I20260620 20:13:47.792303 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:47.792688 24658 external_mini_cluster.cc:1428] Setting key d9cd0f2648122923d555270b84cf2126
I20260620 20:13:47.795833 28790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5: Bootstrap starting.
I20260620 20:13:47.798310 28790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:47.799083 28790 log.cc:824] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:47.801044 28790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5: No bootstrap required, opened a new log
Jun 20 20:13:47 dist-test-slave-pflr krb5kdc[28710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:47.804438 28790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1642648f6efa490385b379084cf83cc5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41663 } }
I20260620 20:13:47.804772 28790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:47.804836 28790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1642648f6efa490385b379084cf83cc5, State: Initialized, Role: FOLLOWER
I20260620 20:13:47.805297 28790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1642648f6efa490385b379084cf83cc5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41663 } }
I20260620 20:13:47.805465 28790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:47.805539 28790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:47.805671 28790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:47.806758 28790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1642648f6efa490385b379084cf83cc5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41663 } }
I20260620 20:13:47.807081 28790 leader_election.cc:302] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1642648f6efa490385b379084cf83cc5; no voters:
I20260620 20:13:47.807403 28790 leader_election.cc:288] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:47.807509 28795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:47.807542 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:47.794038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.24.20.190:41663)
0620 20:13:47.794408 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:47.794432 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:47.794466 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:13:47.795067 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:47.795083 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:47.795389 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:47.795662 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:47.796566 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:47.797582 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:47.798737 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:47.799225 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:47.799350 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:47.802179 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:47.802184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:47.802196 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:47.802221 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:47.804449 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:47.804960 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:47.804966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:47.804971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:47.805142 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:47.805466 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:47.805471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:47.805473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:47.805940 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:47.806102 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:47.806368 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 20:13:47.806656 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":123,"threads_started":1}
I20260620 20:13:47.807750 28795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [term 1 LEADER]: Becoming Leader. State: Replica: 1642648f6efa490385b379084cf83cc5, State: Running, Role: LEADER
I20260620 20:13:47.808043 28795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1642648f6efa490385b379084cf83cc5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41663 } }
I20260620 20:13:47.808530 28790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:47.809187 28796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1642648f6efa490385b379084cf83cc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1642648f6efa490385b379084cf83cc5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41663 } } }
I20260620 20:13:47.809187 28795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1642648f6efa490385b379084cf83cc5. Latest consensus state: current_term: 1 leader_uuid: "1642648f6efa490385b379084cf83cc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1642648f6efa490385b379084cf83cc5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41663 } } }
I20260620 20:13:47.809331 28795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:47.809331 28796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:47.809727 28802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:47.813120 28802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:47.818195 28802 catalog_manager.cc:1347] Generated new cluster ID: cb45633c8f374e4db974eddaad9fa808
I20260620 20:13:47.818260 28802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:47.831898 28802 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:47.833139 28802 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:47.840324 28802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5: Generated new TSK 0
I20260620 20:13:47.841068 28802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:13:47.867662 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41663
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:47.973984 28816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:47.974212 28816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:47.974252 28816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:47.978016 28816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:47.978056 28816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:47.978111 28816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:13:47.983002 28816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:47.983832 28816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:47.984797 28816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:47.996177 28821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:47.996228 28824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:47.996193 28822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:47.996546 28816 server_base.cc:1034] running on GCE node
I20260620 20:13:47.996953 28816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:47.997525 28816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:47.998694 28816 hybrid_clock.cc:648] HybridClock initialized: now 1781986427998674 us; error 31 us; skew 500 ppm
Jun 20 20:13:47 dist-test-slave-pflr krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986427, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:48.001309 28816 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:13:48.001992 28816 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:48.002084 28816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:48.002307 28816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:48.004245 28816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "aa079e023a11449ca0d4b2d8b009466c"
format_stamp: "Formatted at 2026-06-20 20:13:47 on dist-test-slave-pflr"
server_key: "dba9eb57d970efc2dbb19cfd1f3c0bae"
server_key_iv: "ccc9278c019114dcdf8349176df4d3cd"
server_key_version: "encryptionkey@0"
I20260620 20:13:48.004822 28816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "aa079e023a11449ca0d4b2d8b009466c"
format_stamp: "Formatted at 2026-06-20 20:13:47 on dist-test-slave-pflr"
server_key: "dba9eb57d970efc2dbb19cfd1f3c0bae"
server_key_iv: "ccc9278c019114dcdf8349176df4d3cd"
server_key_version: "encryptionkey@0"
I20260620 20:13:48.008311 28816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:13:48.010834 28830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:48.012037 28816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 20:13:48.012179 28816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "aa079e023a11449ca0d4b2d8b009466c"
format_stamp: "Formatted at 2026-06-20 20:13:47 on dist-test-slave-pflr"
server_key: "dba9eb57d970efc2dbb19cfd1f3c0bae"
server_key_iv: "ccc9278c019114dcdf8349176df4d3cd"
server_key_version: "encryptionkey@0"
I20260620 20:13:48.012282 28816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:48.039069 28816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:48.041945 28816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:48.042142 28816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:48.042692 28816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:48.043448 28816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:48.043516 28816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:48.043583 28816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:48.043642 28816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:48.052444 28816 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:44339
I20260620 20:13:48.052464 28943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:44339 every 8 connection(s)
I20260620 20:13:48.053025 28816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:48.053308 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 28816
I20260620 20:13:48.053444 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:13:48.053753 24658 external_mini_cluster.cc:1428] Setting key f183c17df35ac5e8f19bb6d735162184
Jun 20 20:13:48 dist-test-slave-pflr krb5kdc[28710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986427, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
I20260620 20:13:48.066375 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.054714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:56465 (local address 127.24.20.190:41663)
0620 20:13:48.055090 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:48.055094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:48.056083 (+ 989us) server_negotiation.cc:407] Connection header received
0620 20:13:48.057003 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:48.057006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:48.057061 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:48.057158 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:48.058253 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.058785 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.059892 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.060124 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.060238 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.062846 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:48.062849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:48.062851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:48.062873 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:48.064224 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:48.064806 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:48.064810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:48.064814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:48.064923 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:48.065263 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:48.065266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:48.065268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:48.065459 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:48.065572 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:48.066068 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 20:13:48.066203 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303}
I20260620 20:13:48.067286 28946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.055168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:41663 (local address 127.24.20.129:56465)
0620 20:13:48.055951 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:48.055987 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:48.056800 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:48.057322 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:48.057329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:48.057665 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:48.058101 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.058113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.058912 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.058923 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.059764 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.059772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.060272 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.060276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.060436 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.061039 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:48.061054 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:48.062675 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:48.064337 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:48.064343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:48.064356 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:48.064665 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:48.065019 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:48.065023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:48.065025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:48.065162 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:48.065581 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:48.065590 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:48.065819 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 20:13:48.066012 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 20:13:48.066308 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":121,"threads_started":1}
I20260620 20:13:48.068526 28944 heartbeater.cc:347] Connected to a master server at 127.24.20.190:41663
I20260620 20:13:48.068797 28944 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:48.069321 28944 heartbeater.cc:511] Master 127.24.20.190:41663 requested a full tablet report, sending...
I20260620 20:13:48.070787 28754 ts_manager.cc:194] Registered new tserver with Master: aa079e023a11449ca0d4b2d8b009466c (127.24.20.129:44339)
I20260620 20:13:48.071888 28754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:56465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:13:48.081085 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41663
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:48.188465 28949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:48.188730 28949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:48.188776 28949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:48.193395 28949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:48.193463 28949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:48.193564 28949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:13:48.198935 28949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:48.199934 28949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:48.200999 28949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:48.209779 28955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:48.209854 28954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:48.209926 28957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:48.210223 28949 server_base.cc:1034] running on GCE node
I20260620 20:13:48.210683 28949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:48.211316 28949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:48.212517 28949 hybrid_clock.cc:648] HybridClock initialized: now 1781986428212509 us; error 29 us; skew 500 ppm
Jun 20 20:13:48 dist-test-slave-pflr krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986428, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:48.215639 28949 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:13:48.216399 28949 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:48.216459 28949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:48.216672 28949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:48.218546 28949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "8b858cf119c147f28f6582ffe9b0b7a0"
format_stamp: "Formatted at 2026-06-20 20:13:48 on dist-test-slave-pflr"
server_key: "455a4956ef399fc20dd98071db78b2b0"
server_key_iv: "9c5d306c4b83d8b0dd04acd19d31ebbb"
server_key_version: "encryptionkey@0"
I20260620 20:13:48.219030 28949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "8b858cf119c147f28f6582ffe9b0b7a0"
format_stamp: "Formatted at 2026-06-20 20:13:48 on dist-test-slave-pflr"
server_key: "455a4956ef399fc20dd98071db78b2b0"
server_key_iv: "9c5d306c4b83d8b0dd04acd19d31ebbb"
server_key_version: "encryptionkey@0"
I20260620 20:13:48.222452 28949 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 20:13:48.224787 28963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:48.225955 28949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:48.226083 28949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "8b858cf119c147f28f6582ffe9b0b7a0"
format_stamp: "Formatted at 2026-06-20 20:13:48 on dist-test-slave-pflr"
server_key: "455a4956ef399fc20dd98071db78b2b0"
server_key_iv: "9c5d306c4b83d8b0dd04acd19d31ebbb"
server_key_version: "encryptionkey@0"
I20260620 20:13:48.226189 28949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:48.249769 28949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:48.253057 28949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:48.253252 28949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:48.253806 28949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:48.254593 28949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:48.254663 28949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:48.254727 28949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:48.254781 28949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:48.263182 28949 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:34237
I20260620 20:13:48.263217 29076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:34237 every 8 connection(s)
I20260620 20:13:48.263797 28949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:48.266852 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 28949
I20260620 20:13:48.266981 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:13:48.267302 24658 external_mini_cluster.cc:1428] Setting key 6f70637cc513b5e827f3aa5bf152989a
Jun 20 20:13:48 dist-test-slave-pflr krb5kdc[28710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986428, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
I20260620 20:13:48.277928 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.265423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:33193 (local address 127.24.20.190:41663)
0620 20:13:48.265558 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:48.265562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:48.266675 (+ 1113us) server_negotiation.cc:407] Connection header received
0620 20:13:48.267824 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:48.267830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:48.267896 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:48.267963 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:48.269420 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.269936 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.271216 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.271622 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.271759 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.274137 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:48.274143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:48.274147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:48.274182 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:48.275834 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:48.276304 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:48.276310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:48.276315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:48.276405 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:48.276773 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:48.276778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:48.276780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:48.277049 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:48.277165 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:48.277619 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 20:13:48.277733 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:13:48.278656 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.265793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:41663 (local address 127.24.20.130:33193)
0620 20:13:48.266516 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:48.266554 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:48.267589 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:48.268164 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:48.268170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:48.268604 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:48.269267 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.269291 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.270056 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.270063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.271097 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.271106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.271777 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.271780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.271913 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.272538 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:48.272561 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:48.273974 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:48.275945 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:48.275957 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:48.275967 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:48.276197 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:48.276502 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:48.276504 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:48.276505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:48.276684 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:48.277165 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:48.277170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:48.277379 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 20:13:48.277542 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 20:13:48.277770 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
I20260620 20:13:48.279912 29077 heartbeater.cc:347] Connected to a master server at 127.24.20.190:41663
I20260620 20:13:48.280139 29077 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:48.280541 29077 heartbeater.cc:511] Master 127.24.20.190:41663 requested a full tablet report, sending...
I20260620 20:13:48.281646 28754 ts_manager.cc:194] Registered new tserver with Master: 8b858cf119c147f28f6582ffe9b0b7a0 (127.24.20.130:34237)
I20260620 20:13:48.282337 28754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:33193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:13:48.295171 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41663
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:48.403054 29082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:48.403327 29082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:48.403401 29082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:48.407580 29082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:48.407660 29082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:48.407743 29082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:13:48.413172 29082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:48.414153 29082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:48.415117 29082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:48.424335 29088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:48.424372 29090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:48.424340 29087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:48.424530 29082 server_base.cc:1034] running on GCE node
I20260620 20:13:48.425032 29082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:48.425706 29082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:48.426872 29082 hybrid_clock.cc:648] HybridClock initialized: now 1781986428426835 us; error 52 us; skew 500 ppm
Jun 20 20:13:48 dist-test-slave-pflr krb5kdc[28710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986428, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:48.429908 29082 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:13:48.430606 29082 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:48.430666 29082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:48.430824 29082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:48.432864 29082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3"
format_stamp: "Formatted at 2026-06-20 20:13:48 on dist-test-slave-pflr"
server_key: "379b88adb43329514dcc3cfc56cc4cef"
server_key_iv: "24868ce94fdeee4e90dcf1c959f8003c"
server_key_version: "encryptionkey@0"
I20260620 20:13:48.433358 29082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3"
format_stamp: "Formatted at 2026-06-20 20:13:48 on dist-test-slave-pflr"
server_key: "379b88adb43329514dcc3cfc56cc4cef"
server_key_iv: "24868ce94fdeee4e90dcf1c959f8003c"
server_key_version: "encryptionkey@0"
I20260620 20:13:48.436853 29082 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:13:48.439152 29096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:48.440395 29082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:48.440578 29082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3"
format_stamp: "Formatted at 2026-06-20 20:13:48 on dist-test-slave-pflr"
server_key: "379b88adb43329514dcc3cfc56cc4cef"
server_key_iv: "24868ce94fdeee4e90dcf1c959f8003c"
server_key_version: "encryptionkey@0"
I20260620 20:13:48.440753 29082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:48.472791 29082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:48.476197 29082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:48.476400 29082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:48.477005 29082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:48.477830 29082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:48.477905 29082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:48.477975 29082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:48.478020 29082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:48.487017 29082 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:41151
I20260620 20:13:48.487035 29209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:41151 every 8 connection(s)
I20260620 20:13:48.487643 29082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:13:48.491043 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 29082
I20260620 20:13:48.491184 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:13:48.491501 24658 external_mini_cluster.cc:1428] Setting key 1db1a2879e19037b67e616d67ce666c5
Jun 20 20:13:48 dist-test-slave-pflr krb5kdc[28710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986428, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:48.502290 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.489633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:53977 (local address 127.24.20.190:41663)
0620 20:13:48.489808 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:48.489812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:48.491111 (+ 1299us) server_negotiation.cc:407] Connection header received
0620 20:13:48.492155 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:48.492159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:48.492225 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:48.492331 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:48.493787 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.494326 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.495571 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.495813 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.495907 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.498573 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:48.498576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:48.498578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:48.498603 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:48.500152 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:48.500767 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:48.500770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:48.500771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:48.500855 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:48.501210 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:48.501212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:48.501213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:48.501404 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:48.501492 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:48.502051 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 20:13:48.502162 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:13:48.503150 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.490046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:41663 (local address 127.24.20.131:53977)
0620 20:13:48.490945 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:48.490989 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:48.491923 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:48.492519 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:48.492533 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:48.492982 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:48.493607 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.493645 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.494449 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.494453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.495427 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.495436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.495954 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.495957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.496105 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.496759 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:48.496776 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:48.498391 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:48.500271 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:48.500277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:48.500290 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:48.500661 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:48.500956 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:48.500959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:48.500960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:48.501111 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:48.501547 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:48.501553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:48.501799 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 20:13:48.501965 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 20:13:48.502242 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":109,"threads_started":1}
I20260620 20:13:48.504393 29210 heartbeater.cc:347] Connected to a master server at 127.24.20.190:41663
I20260620 20:13:48.504737 29210 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:48.505229 29210 heartbeater.cc:511] Master 127.24.20.190:41663 requested a full tablet report, sending...
I20260620 20:13:48.506211 28754 ts_manager.cc:194] Registered new tserver with Master: 005bc3d4918c41b1a4ae3bdca6ebdba3 (127.24.20.131:41151)
I20260620 20:13:48.506734 28754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:53977
I20260620 20:13:48.517292 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:13:48.526415 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.519246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.24.20.190:41663)
0620 20:13:48.519388 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:48.519394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:48.519517 (+ 123us) server_negotiation.cc:407] Connection header received
0620 20:13:48.519619 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:48.519621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:48.519660 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:48.519729 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:48.520358 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.520980 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.521744 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.521954 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.522030 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.523347 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:48.523350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:48.523352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:48.523372 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:48.524895 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:48.525297 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:48.525299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:48.525300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:48.525369 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:48.525605 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:48.525608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:48.525609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:48.525806 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:48.525909 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:48.526123 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 20:13:48.526208 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:13:48.529462 28754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:13:48.532441 28754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:48.546727 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.540549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:34237 (local address 127.0.0.1:43974)
0620 20:13:48.540855 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:48.541048 (+ 193us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:48.541132 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:48.542035 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:48.542044 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:48.542070 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:48.542330 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.542339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.543712 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.543715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.545176 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.545181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.546176 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.546179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.546291 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.546330 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:13:48.546448 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 20:13:48.546566 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":126,"thread_start_us":103,"threads_started":1}
I20260620 20:13:48.546763 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.541157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43974 (local address 127.24.20.130:34237)
0620 20:13:48.541496 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:48.541512 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:48.541536 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:13:48.541622 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:48.541627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:48.541843 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:48.542037 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:48.542475 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.543584 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.545278 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.546064 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.546146 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.546250 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:48.546509 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 20:13:48.546601 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":122,"threads_started":1}
I20260620 20:13:48.547616 29221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.540323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:41151 (local address 127.0.0.1:60004)
0620 20:13:48.540839 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:48.540884 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:48.541019 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:48.542082 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:48.542084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:48.542097 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:48.542341 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.542353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.543467 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.543471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.545170 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.545180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.547225 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.547232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.547330 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.547353 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:13:48.547403 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 20:13:48.547470 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":228,"thread_start_us":76,"threads_started":1}
I20260620 20:13:48.547683 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.541458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.24.20.129:44339)
0620 20:13:48.541869 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:48.541875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:48.541931 (+ 56us) server_negotiation.cc:407] Connection header received
0620 20:13:48.542122 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:48.542128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:48.542296 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:48.542458 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:48.542864 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.543908 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.545907 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.546631 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.546724 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.546882 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:48.547400 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 20:13:48.547545 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":90,"threads_started":1}
I20260620 20:13:48.547971 29224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.541064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:44339 (local address 127.0.0.1:48354)
0620 20:13:48.541849 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:48.541862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:48.541974 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:48.542624 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:48.542627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:48.542644 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:48.542741 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.542747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.544030 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.544033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.545777 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.545787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.546945 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.546949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.547040 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.547062 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:13:48.547128 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 20:13:48.547850 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":630,"thread_start_us":80,"threads_started":1}
I20260620 20:13:48.548254 29222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.540468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.24.20.131:41151)
0620 20:13:48.541619 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:48.541632 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:48.541656 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:13:48.541733 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:48.541738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:48.541935 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:48.542250 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:48.542509 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.543336 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.546297 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.547084 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.547794 (+ 710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.547969 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:48.548033 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 20:13:48.548114 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":101,"threads_started":1}
I20260620 20:13:48.550110 29011 tablet_service.cc:1467] Processing CreateTablet for tablet a21563a21543415abda8ad706fec1bce (DEFAULT_TABLE table=test-table [id=916131a1e1334e478a43580b384fe60f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:48.550693 28878 tablet_service.cc:1467] Processing CreateTablet for tablet a21563a21543415abda8ad706fec1bce (DEFAULT_TABLE table=test-table [id=916131a1e1334e478a43580b384fe60f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:48.550693 29144 tablet_service.cc:1467] Processing CreateTablet for tablet a21563a21543415abda8ad706fec1bce (DEFAULT_TABLE table=test-table [id=916131a1e1334e478a43580b384fe60f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:48.551124 29011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21563a21543415abda8ad706fec1bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:48.551677 28878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21563a21543415abda8ad706fec1bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:48.551681 29144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21563a21543415abda8ad706fec1bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:48.557785 29227 tablet_bootstrap.cc:492] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0: Bootstrap starting.
I20260620 20:13:48.559077 29229 tablet_bootstrap.cc:492] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3: Bootstrap starting.
I20260620 20:13:48.559556 29228 tablet_bootstrap.cc:492] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c: Bootstrap starting.
I20260620 20:13:48.560039 29227 tablet_bootstrap.cc:654] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:48.560810 29227 log.cc:824] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:48.561086 29229 tablet_bootstrap.cc:654] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:48.561553 29228 tablet_bootstrap.cc:654] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:48.562315 29229 log.cc:824] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:48.562690 29228 log.cc:824] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:48.562834 29227 tablet_bootstrap.cc:492] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0: No bootstrap required, opened a new log
I20260620 20:13:48.563035 29227 ts_tablet_manager.cc:1397] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 20:13:48.564375 29228 tablet_bootstrap.cc:492] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c: No bootstrap required, opened a new log
I20260620 20:13:48.564384 29229 tablet_bootstrap.cc:492] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3: No bootstrap required, opened a new log
I20260620 20:13:48.564566 29229 ts_tablet_manager.cc:1397] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:13:48.564580 29228 ts_tablet_manager.cc:1397] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:13:48.566983 29227 raft_consensus.cc:348] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41151 } } peers { permanent_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34237 } } peers { permanent_uuid: "aa079e023a11449ca0d4b2d8b009466c" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44339 } }
I20260620 20:13:48.567294 29227 raft_consensus.cc:374] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:48.567381 29227 raft_consensus.cc:729] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b858cf119c147f28f6582ffe9b0b7a0, State: Initialized, Role: FOLLOWER
I20260620 20:13:48.567533 29228 raft_consensus.cc:348] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41151 } } peers { permanent_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34237 } } peers { permanent_uuid: "aa079e023a11449ca0d4b2d8b009466c" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44339 } }
I20260620 20:13:48.567942 29228 raft_consensus.cc:374] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:48.567886 29227 consensus_queue.cc:260] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41151 } } peers { permanent_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34237 } } peers { permanent_uuid: "aa079e023a11449ca0d4b2d8b009466c" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44339 } }
I20260620 20:13:48.568013 29228 raft_consensus.cc:729] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa079e023a11449ca0d4b2d8b009466c, State: Initialized, Role: FOLLOWER
I20260620 20:13:48.568300 29229 raft_consensus.cc:348] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41151 } } peers { permanent_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34237 } } peers { permanent_uuid: "aa079e023a11449ca0d4b2d8b009466c" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44339 } }
I20260620 20:13:48.568536 29229 raft_consensus.cc:374] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:48.568562 29077 heartbeater.cc:503] Master 127.24.20.190:41663 was elected leader, sending a full tablet report...
I20260620 20:13:48.568607 29229 raft_consensus.cc:729] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 005bc3d4918c41b1a4ae3bdca6ebdba3, State: Initialized, Role: FOLLOWER
I20260620 20:13:48.568984 29227 ts_tablet_manager.cc:1428] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:13:48.568986 29229 consensus_queue.cc:260] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41151 } } peers { permanent_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34237 } } peers { permanent_uuid: "aa079e023a11449ca0d4b2d8b009466c" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44339 } }
I20260620 20:13:48.569653 29210 heartbeater.cc:503] Master 127.24.20.190:41663 was elected leader, sending a full tablet report...
I20260620 20:13:48.569644 29228 consensus_queue.cc:260] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41151 } } peers { permanent_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34237 } } peers { permanent_uuid: "aa079e023a11449ca0d4b2d8b009466c" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44339 } }
I20260620 20:13:48.569898 29229 ts_tablet_manager.cc:1428] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:13:48.570499 29228 ts_tablet_manager.cc:1428] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260620 20:13:48.570863 28944 heartbeater.cc:503] Master 127.24.20.190:41663 was elected leader, sending a full tablet report...
I20260620 20:13:48.688345 29233 raft_consensus.cc:482] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:48.688730 29233 raft_consensus.cc:504] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41151 } } peers { permanent_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34237 } } peers { permanent_uuid: "aa079e023a11449ca0d4b2d8b009466c" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44339 } }
I20260620 20:13:48.690081 29233 leader_election.cc:288] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 005bc3d4918c41b1a4ae3bdca6ebdba3 (127.24.20.131:41151), aa079e023a11449ca0d4b2d8b009466c (127.24.20.129:44339)
I20260620 20:13:48.693763 29222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.690479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:33821 (local address 127.24.20.131:41151)
0620 20:13:48.690628 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:48.690636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:48.690656 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:13:48.690718 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:48.690721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:48.690774 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:48.690879 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:48.691184 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.691690 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.692745 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.693158 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.693330 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.693386 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:48.693577 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 20:13:48.693640 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:13:48.693763 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.690286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:41151 (local address 127.24.20.130:33821)
0620 20:13:48.690411 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:48.690428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:48.690525 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:48.690905 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:48.690907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:48.690924 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:48.691034 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.691039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.691805 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.691811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.692586 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.692621 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.693318 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.693322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.693429 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.693452 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:13:48.693564 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 20:13:48.693636 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 20:13:48.694417 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.690407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:44339 (local address 127.24.20.130:54493)
0620 20:13:48.690829 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:48.690845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:48.691001 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:48.691302 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:48.691306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:48.691323 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:48.691452 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.691457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.692235 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.692240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.693159 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:48.693168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.693989 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.693995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:48.694102 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.694126 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:13:48.694193 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 20:13:48.694272 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":139,"threads_started":1}
I20260620 20:13:48.694543 29164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21563a21543415abda8ad706fec1bce" candidate_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3" is_pre_election: true
I20260620 20:13:48.694839 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.690579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:54493 (local address 127.24.20.129:44339)
0620 20:13:48.690718 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:48.690724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:48.690915 (+ 191us) server_negotiation.cc:407] Connection header received
0620 20:13:48.691151 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:48.691155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:48.691199 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:48.691290 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:48.691578 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.692112 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.693293 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.693737 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.693884 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.694613 (+ 729us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:48.694654 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 20:13:48.694723 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:13:48.694888 29164 raft_consensus.cc:2454] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b858cf119c147f28f6582ffe9b0b7a0 in term 0.
I20260620 20:13:48.695390 28964 leader_election.cc:302] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 005bc3d4918c41b1a4ae3bdca6ebdba3, 8b858cf119c147f28f6582ffe9b0b7a0; no voters:
I20260620 20:13:48.695379 28898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21563a21543415abda8ad706fec1bce" candidate_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa079e023a11449ca0d4b2d8b009466c" is_pre_election: true
I20260620 20:13:48.695689 28898 raft_consensus.cc:2454] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b858cf119c147f28f6582ffe9b0b7a0 in term 0.
I20260620 20:13:48.695690 29233 raft_consensus.cc:2781] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:48.695786 29233 raft_consensus.cc:482] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:48.695840 29233 raft_consensus.cc:3037] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:48.697137 29233 raft_consensus.cc:504] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41151 } } peers { permanent_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34237 } } peers { permanent_uuid: "aa079e023a11449ca0d4b2d8b009466c" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44339 } }
I20260620 20:13:48.697516 29233 leader_election.cc:288] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [CANDIDATE]: Term 1 election: Requested vote from peers 005bc3d4918c41b1a4ae3bdca6ebdba3 (127.24.20.131:41151), aa079e023a11449ca0d4b2d8b009466c (127.24.20.129:44339)
I20260620 20:13:48.697809 29164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21563a21543415abda8ad706fec1bce" candidate_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3"
I20260620 20:13:48.697937 29164 raft_consensus.cc:3037] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:48.697935 28898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21563a21543415abda8ad706fec1bce" candidate_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa079e023a11449ca0d4b2d8b009466c"
I20260620 20:13:48.698076 28898 raft_consensus.cc:3037] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:48.698990 29164 raft_consensus.cc:2454] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b858cf119c147f28f6582ffe9b0b7a0 in term 1.
I20260620 20:13:48.699028 28898 raft_consensus.cc:2454] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b858cf119c147f28f6582ffe9b0b7a0 in term 1.
I20260620 20:13:48.699280 28964 leader_election.cc:302] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 005bc3d4918c41b1a4ae3bdca6ebdba3, 8b858cf119c147f28f6582ffe9b0b7a0; no voters:
I20260620 20:13:48.699482 29233 raft_consensus.cc:2781] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:48.699692 29233 raft_consensus.cc:686] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [term 1 LEADER]: Becoming Leader. State: Replica: 8b858cf119c147f28f6582ffe9b0b7a0, State: Running, Role: LEADER
I20260620 20:13:48.699998 29233 consensus_queue.cc:237] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41151 } } peers { permanent_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34237 } } peers { permanent_uuid: "aa079e023a11449ca0d4b2d8b009466c" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44339 } }
I20260620 20:13:48.703193 28752 catalog_manager.cc:5495] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b858cf119c147f28f6582ffe9b0b7a0 (127.24.20.130). New cstate: current_term: 1 leader_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b858cf119c147f28f6582ffe9b0b7a0" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa079e023a11449ca0d4b2d8b009466c" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44339 } health_report { overall_health: UNKNOWN } } }
W20260620 20:13:48.738580 29211 tablet.cc:2369] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:13:48.742558 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.738902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.24.20.130:34237)
0620 20:13:48.739061 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:48.739065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:48.739083 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:13:48.739157 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:48.739160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:48.739214 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:48.739307 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:48.739549 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.740157 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.741079 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:48.741331 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:48.741419 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:48.741698 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:48.742128 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:48.742222 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:48.742295 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 20:13:48.742346 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:13:48.748490 28898 raft_consensus.cc:1261] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c [term 1 FOLLOWER]: Refusing update from remote peer 8b858cf119c147f28f6582ffe9b0b7a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:48.748490 29164 raft_consensus.cc:1261] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3 [term 1 FOLLOWER]: Refusing update from remote peer 8b858cf119c147f28f6582ffe9b0b7a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:48.749231 29237 consensus_queue.cc:1035] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa079e023a11449ca0d4b2d8b009466c" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:48.749406 29233 consensus_queue.cc:1035] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "005bc3d4918c41b1a4ae3bdca6ebdba3" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:48.757241 29242 mvcc.cc:205] Tried to move back new op lower bound from 7299016412146642944 to 7299016411957723136. Current Snapshot: MvccSnapshot[applied={T|T < 7299016412146642944}]
I20260620 20:13:48.757582 29241 mvcc.cc:205] Tried to move back new op lower bound from 7299016412146642944 to 7299016411957723136. Current Snapshot: MvccSnapshot[applied={T|T < 7299016412146642944}]
I20260620 20:13:48.767076 28753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:48.767249 28753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:48.769484 28753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1642648f6efa490385b379084cf83cc5: Sending DeleteTablet for 3 replicas of tablet a21563a21543415abda8ad706fec1bce
I20260620 20:13:48.770224 29144 tablet_service.cc:1514] Processing DeleteTablet for tablet a21563a21543415abda8ad706fec1bce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:48 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:60004
I20260620 20:13:48.770256 28878 tablet_service.cc:1514] Processing DeleteTablet for tablet a21563a21543415abda8ad706fec1bce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:48 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:48354
I20260620 20:13:48.770690 29250 tablet_replica.cc:330] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3: stopping tablet replica
I20260620 20:13:48.770685 29011 tablet_service.cc:1514] Processing DeleteTablet for tablet a21563a21543415abda8ad706fec1bce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:48 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:43974
I20260620 20:13:48.770992 29250 raft_consensus.cc:2229] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:48.771173 29252 tablet_replica.cc:330] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0: stopping tablet replica
I20260620 20:13:48.771302 29250 raft_consensus.cc:2258] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:48.771529 29252 raft_consensus.cc:2229] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:13:48.771737 29251 tablet_replica.cc:330] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c: stopping tablet replica
I20260620 20:13:48.771839 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 28816
I20260620 20:13:48.772025 29251 raft_consensus.cc:2229] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:48.772279 29251 raft_consensus.cc:2258] T a21563a21543415abda8ad706fec1bce P aa079e023a11449ca0d4b2d8b009466c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:48.773110 29252 raft_consensus.cc:2258] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:48.773269 29250 ts_tablet_manager.cc:1905] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:48.774333 29252 ts_tablet_manager.cc:1905] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:48.775923 29250 ts_tablet_manager.cc:1918] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:48.776023 29250 log.cc:1196] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/a21563a21543415abda8ad706fec1bce
I20260620 20:13:48.776355 29250 ts_tablet_manager.cc:1939] T a21563a21543415abda8ad706fec1bce P 005bc3d4918c41b1a4ae3bdca6ebdba3: Deleting consensus metadata
I20260620 20:13:48.777084 29252 ts_tablet_manager.cc:1918] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:48.777179 29252 log.cc:1196] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/a21563a21543415abda8ad706fec1bce
I20260620 20:13:48.777501 29252 ts_tablet_manager.cc:1939] T a21563a21543415abda8ad706fec1bce P 8b858cf119c147f28f6582ffe9b0b7a0: Deleting consensus metadata
I20260620 20:13:48.777629 28738 catalog_manager.cc:4841] TS 005bc3d4918c41b1a4ae3bdca6ebdba3 (127.24.20.131:41151): tablet a21563a21543415abda8ad706fec1bce (table test-table [id=916131a1e1334e478a43580b384fe60f]) successfully deleted
I20260620 20:13:48.778461 28740 catalog_manager.cc:4841] TS 8b858cf119c147f28f6582ffe9b0b7a0 (127.24.20.130:34237): tablet a21563a21543415abda8ad706fec1bce (table test-table [id=916131a1e1334e478a43580b384fe60f]) successfully deleted
W20260620 20:13:48.780094 28740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:44339 (error 108)
I20260620 20:13:48.780923 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 28949
I20260620 20:13:48.781152 29224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.780829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:44339 (local address 127.0.0.1:48370)
0620 20:13:48.780984 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:48.781059 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:44339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 20:13:48.781613 28740 catalog_manager.cc:4584] TS aa079e023a11449ca0d4b2d8b009466c (127.24.20.129:44339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a21563a21543415abda8ad706fec1bce: Network error: Client connection negotiation failed: client connection to 127.24.20.129:44339: connect: Connection refused (error 111)
I20260620 20:13:48.787199 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 29082
I20260620 20:13:48.792678 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 28723
2026-06-20T20:13:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2329 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T20:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:48Z Disabled control of system clock
I20260620 20:13:48.823223 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:43121
--webserver_interface=127.24.20.190
--webserver_port=0
--builtin_ntp_servers=127.24.20.148:39059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:43121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 20:13:48.930191 29257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:48.930426 29257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:48.930502 29257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:48.934394 29257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:48.934448 29257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:48.934463 29257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:48.934480 29257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:48.934531 29257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:48.939275 29257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:39059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:43121
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:43121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:48.940271 29257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:48.941591 29257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:48.949699 29265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:48.949741 29262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:48.949801 29257 server_base.cc:1034] running on GCE node
W20260620 20:13:48.949753 29263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:48.950484 29257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:48.951472 29257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:48.952669 29257 hybrid_clock.cc:648] HybridClock initialized: now 1781986428952643 us; error 30 us; skew 500 ppm
I20260620 20:13:48.957484 29257 webserver.cc:460] Webserver started at http://127.24.20.190:37087/ using document root <none> and password file <none>
I20260620 20:13:48.958040 29257 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:48.958101 29257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:48.958371 29257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:48.960325 29257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "1cc09970e78c4533b017e29b2255c43e"
format_stamp: "Formatted at 2026-06-20 20:13:48 on dist-test-slave-pflr"
server_key: "fd5d33a4a28dcc4acc957946f415349b"
server_key_iv: "f1c190b672e93460e98aff8404cc673b"
server_key_version: "encryptionkey@0"
I20260620 20:13:48.960891 29257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "1cc09970e78c4533b017e29b2255c43e"
format_stamp: "Formatted at 2026-06-20 20:13:48 on dist-test-slave-pflr"
server_key: "fd5d33a4a28dcc4acc957946f415349b"
server_key_iv: "f1c190b672e93460e98aff8404cc673b"
server_key_version: "encryptionkey@0"
I20260620 20:13:48.964288 29257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:13:48.967012 29271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:48.968174 29257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:48.968305 29257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "1cc09970e78c4533b017e29b2255c43e"
format_stamp: "Formatted at 2026-06-20 20:13:48 on dist-test-slave-pflr"
server_key: "fd5d33a4a28dcc4acc957946f415349b"
server_key_iv: "f1c190b672e93460e98aff8404cc673b"
server_key_version: "encryptionkey@0"
I20260620 20:13:48.968410 29257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:48.972891 29257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:48.973445 29257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:48.973621 29257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:48.979949 29257 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:43121
I20260620 20:13:48.979954 29323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:43121 every 8 connection(s)
I20260620 20:13:48.980918 29257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:48.983777 29324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:48.988853 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 29257
I20260620 20:13:48.988973 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:48.989037 29324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e: Bootstrap starting.
I20260620 20:13:48.989284 24658 external_mini_cluster.cc:1428] Setting key d777198e88a7e660e6bf536cde3f1eb1
I20260620 20:13:48.991494 29324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:48.992192 29324 log.cc:824] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:48.994318 29324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e: No bootstrap required, opened a new log
I20260620 20:13:48.994511 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:48.990503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.24.20.190:43121)
0620 20:13:48.991138 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:48.991154 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:48.991186 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:13:48.991832 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:48.991850 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:48.992184 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:48.992494 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:48.993294 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:48.993298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:48.993309 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:48.993317 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:48.993365 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:48.993380 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:48.993463 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:48.993591 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 20:13:48.993826 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":135,"threads_started":1}
I20260620 20:13:48.998258 29324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc09970e78c4533b017e29b2255c43e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 43121 } }
I20260620 20:13:48.998476 29324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:48.998533 29324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc09970e78c4533b017e29b2255c43e, State: Initialized, Role: FOLLOWER
I20260620 20:13:48.998894 29324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc09970e78c4533b017e29b2255c43e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 43121 } }
I20260620 20:13:48.999043 29324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:48.999135 29324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:48.999233 29324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:49.000186 29324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc09970e78c4533b017e29b2255c43e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 43121 } }
I20260620 20:13:49.000504 29324 leader_election.cc:302] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cc09970e78c4533b017e29b2255c43e; no voters:
I20260620 20:13:49.000801 29324 leader_election.cc:288] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:49.000885 29329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:49.001157 29329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [term 1 LEADER]: Becoming Leader. State: Replica: 1cc09970e78c4533b017e29b2255c43e, State: Running, Role: LEADER
I20260620 20:13:49.001443 29329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc09970e78c4533b017e29b2255c43e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 43121 } }
I20260620 20:13:49.001950 29324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:49.002872 29331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc09970e78c4533b017e29b2255c43e. Latest consensus state: current_term: 1 leader_uuid: "1cc09970e78c4533b017e29b2255c43e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc09970e78c4533b017e29b2255c43e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 43121 } } }
I20260620 20:13:49.003000 29331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:49.003113 29330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cc09970e78c4533b017e29b2255c43e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc09970e78c4533b017e29b2255c43e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 43121 } } }
I20260620 20:13:49.003193 29330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:49.003480 29338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:49.006412 29338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:49.011435 29338 catalog_manager.cc:1347] Generated new cluster ID: 2c56aa39cfa448949c5fdbd3342f3351
I20260620 20:13:49.011494 29338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:49.021178 29338 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:49.021450 29338 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:49.026896 29338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e: Generated new TSK 0
I20260620 20:13:49.027493 29338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:13:49.032518 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:43121
--builtin_ntp_servers=127.24.20.148:39059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 20:13:49.145292 29348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:49.145524 29348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:49.145560 29348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:49.149925 29348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:49.149978 29348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:49.150032 29348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:13:49.154716 29348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:39059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:49.155687 29348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:49.157063 29348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:49.166292 29353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:49.166344 29354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:49.166338 29348 server_base.cc:1034] running on GCE node
W20260620 20:13:49.166548 29356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:49.166920 29348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:49.167625 29348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:49.170895 29348 hybrid_clock.cc:648] HybridClock initialized: now 1781986429170875 us; error 28 us; skew 500 ppm
I20260620 20:13:49.172803 29348 webserver.cc:460] Webserver started at http://127.24.20.129:43803/ using document root <none> and password file <none>
I20260620 20:13:49.173357 29348 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:49.173415 29348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:49.173662 29348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:49.175617 29348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "014e29e9a420412e8b539b2779251e78"
format_stamp: "Formatted at 2026-06-20 20:13:49 on dist-test-slave-pflr"
server_key: "8942ff5edcb166f90356663013640968"
server_key_iv: "02e54a9b4db7fcace09d7625220c0fd5"
server_key_version: "encryptionkey@0"
I20260620 20:13:49.176168 29348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "014e29e9a420412e8b539b2779251e78"
format_stamp: "Formatted at 2026-06-20 20:13:49 on dist-test-slave-pflr"
server_key: "8942ff5edcb166f90356663013640968"
server_key_iv: "02e54a9b4db7fcace09d7625220c0fd5"
server_key_version: "encryptionkey@0"
I20260620 20:13:49.179708 29348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:13:49.182081 29362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:49.183233 29348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:49.183362 29348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "014e29e9a420412e8b539b2779251e78"
format_stamp: "Formatted at 2026-06-20 20:13:49 on dist-test-slave-pflr"
server_key: "8942ff5edcb166f90356663013640968"
server_key_iv: "02e54a9b4db7fcace09d7625220c0fd5"
server_key_version: "encryptionkey@0"
I20260620 20:13:49.183468 29348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:49.187844 29348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:49.188446 29348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:49.188685 29348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:49.189182 29348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:49.190018 29348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:49.190088 29348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:49.190150 29348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:49.190202 29348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:49.199002 29348 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:45617
I20260620 20:13:49.199025 29475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:45617 every 8 connection(s)
I20260620 20:13:49.199834 29348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:49.205863 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.201530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:38667 (local address 127.24.20.190:43121)
0620 20:13:49.201664 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:49.201668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:49.202789 (+ 1121us) server_negotiation.cc:407] Connection header received
0620 20:13:49.204184 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:49.204188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:49.204235 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:49.204335 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:49.205382 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:49.205386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:49.205389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:49.205400 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:49.205424 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:49.205437 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:49.205537 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:49.205687 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 20:13:49.205731 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:13:49.206626 29478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.201931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:43121 (local address 127.24.20.129:38667)
0620 20:13:49.202617 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:49.202682 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:49.203971 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:49.204493 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:49.204500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:49.204582 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:49.205143 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:13:49.205144 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:49.205181 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:13:49.205244 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:49.205519 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:49.205527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:49.205532 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:13:49.205727 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 20:13:49.205949 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260620 20:13:49.207762 29476 heartbeater.cc:347] Connected to a master server at 127.24.20.190:43121
I20260620 20:13:49.207973 29476 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:49.208344 29476 heartbeater.cc:511] Master 127.24.20.190:43121 requested a full tablet report, sending...
I20260620 20:13:49.208873 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 29348
I20260620 20:13:49.209007 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:13:49.209360 24658 external_mini_cluster.cc:1428] Setting key a368d574f69b4cd3297c4c1a394e2342
I20260620 20:13:49.210145 29288 ts_manager.cc:194] Registered new tserver with Master: 014e29e9a420412e8b539b2779251e78 (127.24.20.129:45617)
I20260620 20:13:49.211467 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:43121
--builtin_ntp_servers=127.24.20.148:39059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 20:13:49.319240 29479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:49.319499 29479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:49.319599 29479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:49.323630 29479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:49.323705 29479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:49.323787 29479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:13:49.328334 29479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:39059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:49.329335 29479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:49.330621 29479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:49.339576 29484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:49.339572 29487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:49.339636 29485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:49.340016 29479 server_base.cc:1034] running on GCE node
I20260620 20:13:49.340349 29479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:49.340996 29479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:49.342165 29479 hybrid_clock.cc:648] HybridClock initialized: now 1781986429342142 us; error 35 us; skew 500 ppm
I20260620 20:13:49.343815 29479 webserver.cc:460] Webserver started at http://127.24.20.130:33289/ using document root <none> and password file <none>
I20260620 20:13:49.344326 29479 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:49.344384 29479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:49.344537 29479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:49.346494 29479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "ed85c14b4ef040a5b865c38063cffc96"
format_stamp: "Formatted at 2026-06-20 20:13:49 on dist-test-slave-pflr"
server_key: "5a9b8904520e12499c4e5811cfdb49a4"
server_key_iv: "9d95a7e8f0f7259f890f6269616e19d0"
server_key_version: "encryptionkey@0"
I20260620 20:13:49.346993 29479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "ed85c14b4ef040a5b865c38063cffc96"
format_stamp: "Formatted at 2026-06-20 20:13:49 on dist-test-slave-pflr"
server_key: "5a9b8904520e12499c4e5811cfdb49a4"
server_key_iv: "9d95a7e8f0f7259f890f6269616e19d0"
server_key_version: "encryptionkey@0"
I20260620 20:13:49.350512 29479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 20:13:49.352887 29493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:49.354012 29479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:13:49.354125 29479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "ed85c14b4ef040a5b865c38063cffc96"
format_stamp: "Formatted at 2026-06-20 20:13:49 on dist-test-slave-pflr"
server_key: "5a9b8904520e12499c4e5811cfdb49a4"
server_key_iv: "9d95a7e8f0f7259f890f6269616e19d0"
server_key_version: "encryptionkey@0"
I20260620 20:13:49.354234 29479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:49.358557 29479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:49.359100 29479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:49.359275 29479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:49.359774 29479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:49.360635 29479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:49.360692 29479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:49.360759 29479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:49.360785 29479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:49.369212 29479 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:40095
I20260620 20:13:49.369239 29606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:40095 every 8 connection(s)
I20260620 20:13:49.370035 29479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:49.375828 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.371799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:36639 (local address 127.24.20.190:43121)
0620 20:13:49.371947 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:49.371951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:49.373059 (+ 1108us) server_negotiation.cc:407] Connection header received
0620 20:13:49.374347 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:49.374351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:49.374385 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:49.374459 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:49.375458 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:49.375461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:49.375463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:49.375473 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:49.375491 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:49.375499 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:49.375572 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:49.375692 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 20:13:49.375727 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:13:49.376720 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.372200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:43121 (local address 127.24.20.130:36639)
0620 20:13:49.372892 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:49.372955 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:49.374174 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:49.374635 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:49.374641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:49.374726 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:49.375219 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:13:49.375221 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:49.375266 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:13:49.375329 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:49.375572 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:49.375578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:49.375582 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:13:49.375705 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 20:13:49.375898 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
I20260620 20:13:49.376941 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 29479
I20260620 20:13:49.377074 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:13:49.377390 24658 external_mini_cluster.cc:1428] Setting key 70b1a32e78243863b664723be5f1638e
I20260620 20:13:49.377847 29607 heartbeater.cc:347] Connected to a master server at 127.24.20.190:43121
I20260620 20:13:49.378068 29607 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:49.378455 29607 heartbeater.cc:511] Master 127.24.20.190:43121 requested a full tablet report, sending...
I20260620 20:13:49.379376 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:43121
--builtin_ntp_servers=127.24.20.148:39059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 20:13:49.379525 29288 ts_manager.cc:194] Registered new tserver with Master: ed85c14b4ef040a5b865c38063cffc96 (127.24.20.130:40095)
W20260620 20:13:49.485702 29610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:49.485965 29610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:49.486034 29610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:49.490178 29610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:49.490250 29610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:49.490339 29610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:13:49.494958 29610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:39059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:49.496127 29610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:49.497478 29610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:49.506793 29616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:49.506788 29615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:49.506788 29618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:49.507192 29610 server_base.cc:1034] running on GCE node
I20260620 20:13:49.507572 29610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:49.508164 29610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:49.509320 29610 hybrid_clock.cc:648] HybridClock initialized: now 1781986429509308 us; error 22 us; skew 500 ppm
I20260620 20:13:49.511050 29610 webserver.cc:460] Webserver started at http://127.24.20.131:35783/ using document root <none> and password file <none>
I20260620 20:13:49.511615 29610 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:49.511678 29610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:49.511884 29610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:49.513924 29610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "022ea45a521542bebed7b91864db5f11"
format_stamp: "Formatted at 2026-06-20 20:13:49 on dist-test-slave-pflr"
server_key: "8145579533c648332d81bc7ed1cd9cad"
server_key_iv: "e36e772685fb15e9c37f3f783f57be91"
server_key_version: "encryptionkey@0"
I20260620 20:13:49.514470 29610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "022ea45a521542bebed7b91864db5f11"
format_stamp: "Formatted at 2026-06-20 20:13:49 on dist-test-slave-pflr"
server_key: "8145579533c648332d81bc7ed1cd9cad"
server_key_iv: "e36e772685fb15e9c37f3f783f57be91"
server_key_version: "encryptionkey@0"
I20260620 20:13:49.518043 29610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:13:49.520499 29624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:49.521755 29610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:49.521869 29610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "022ea45a521542bebed7b91864db5f11"
format_stamp: "Formatted at 2026-06-20 20:13:49 on dist-test-slave-pflr"
server_key: "8145579533c648332d81bc7ed1cd9cad"
server_key_iv: "e36e772685fb15e9c37f3f783f57be91"
server_key_version: "encryptionkey@0"
I20260620 20:13:49.521979 29610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:49.526409 29610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:49.526989 29610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:49.527174 29610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:49.527678 29610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:49.528541 29610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:49.528637 29610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:49.528723 29610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:49.528776 29610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:49.537250 29610 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:42717
I20260620 20:13:49.537279 29737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:42717 every 8 connection(s)
I20260620 20:13:49.538086 29610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:13:49.544127 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.540046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:39049 (local address 127.24.20.190:43121)
0620 20:13:49.540214 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:49.540218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:49.541331 (+ 1113us) server_negotiation.cc:407] Connection header received
0620 20:13:49.542600 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:49.542602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:49.542641 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:49.542726 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:49.543746 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:49.543749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:49.543751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:49.543761 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:49.543778 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:49.543785 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:49.543861 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:49.543981 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 20:13:49.544016 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:13:49.545046 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.540482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:43121 (local address 127.24.20.131:39049)
0620 20:13:49.541182 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:49.541222 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:49.542366 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:49.542882 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:49.542889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:49.542972 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:49.543482 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:13:49.543483 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:49.543520 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:13:49.543592 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:49.543859 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:49.543865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:49.543869 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:13:49.543998 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 20:13:49.544227 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":160,"threads_started":1}
I20260620 20:13:49.545344 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 29610
I20260620 20:13:49.545467 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:13:49.545740 24658 external_mini_cluster.cc:1428] Setting key ab6f7dbf19ec621907ab9654fbe7b687
I20260620 20:13:49.546214 29738 heartbeater.cc:347] Connected to a master server at 127.24.20.190:43121
I20260620 20:13:49.546454 29738 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:49.546916 29738 heartbeater.cc:511] Master 127.24.20.190:43121 requested a full tablet report, sending...
I20260620 20:13:49.547938 29288 ts_manager.cc:194] Registered new tserver with Master: 022ea45a521542bebed7b91864db5f11 (127.24.20.131:42717)
I20260620 20:13:49.559355 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:13:49.562974 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.561311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.24.20.190:43121)
0620 20:13:49.561536 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:49.561541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:49.561643 (+ 102us) server_negotiation.cc:407] Connection header received
0620 20:13:49.561795 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:49.561798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:49.561840 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:49.561934 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:49.562604 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:49.562607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:49.562610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:49.562621 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:49.562639 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:49.562648 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:49.562743 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:49.562816 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 20:13:49.562856 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 20:13:49.565559 29288 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:13:49.568430 29288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:49.579448 29749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.576189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:40095 (local address 127.0.0.1:58098)
0620 20:13:49.576532 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:49.576578 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:49.576706 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:49.577984 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:49.577990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:49.578068 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:49.578650 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:13:49.578654 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:49.578676 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:13:49.578743 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:49.579098 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:49.579108 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:49.579114 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:13:49.579233 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 20:13:49.579298 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":76,"threads_started":1}
I20260620 20:13:49.579517 29750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.576336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58098 (local address 127.24.20.130:40095)
0620 20:13:49.576845 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:49.576859 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:49.576883 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:13:49.576937 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:49.576942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:49.577143 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:49.577282 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:49.578893 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:49.578900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:49.578912 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:49.578925 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:49.578970 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:49.579005 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:49.579089 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:49.579274 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 20:13:49.579352 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":111,"threads_started":1}
I20260620 20:13:49.579833 29752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.576913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:45617 (local address 127.0.0.1:59502)
0620 20:13:49.577683 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:49.577697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:49.577767 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:49.578364 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:49.578367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:49.578380 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:49.579146 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:13:49.579148 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:49.579155 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:13:49.579174 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:49.579572 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:49.579575 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:49.579580 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:13:49.579654 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:13:49.579692 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":95,"threads_started":1}
I20260620 20:13:49.579944 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.576988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.24.20.131:42717)
0620 20:13:49.577625 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:49.577642 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:49.577666 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:13:49.577717 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:49.577723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:49.577885 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:49.578034 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:49.579228 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:49.579232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:49.579244 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:49.579255 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:49.579293 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:49.579315 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:49.579402 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:49.579736 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 20:13:49.579801 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":93,"threads_started":1}
I20260620 20:13:49.580338 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.577298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.24.20.129:45617)
0620 20:13:49.578004 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:49.578015 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:49.578035 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:13:49.578079 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:49.578083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:49.578254 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:49.578380 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:49.579321 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:49.579328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:49.579338 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:49.579349 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:49.579392 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:49.579420 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:49.579527 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:49.580135 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 20:13:49.580203 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":126,"threads_started":1}
I20260620 20:13:49.580225 29751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.576784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:42717 (local address 127.0.0.1:36460)
0620 20:13:49.577185 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:49.577199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:49.577269 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:49.578020 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:49.578023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:49.578068 (+ 45us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:49.578704 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:13:49.578707 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:49.578719 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:13:49.578752 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:49.579440 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:49.579443 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:49.579446 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 20:13:49.580060 (+ 614us) client_negotiation.cc:240] Negotiation successful
0620 20:13:49.580099 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":87,"threads_started":1}
I20260620 20:13:49.582449 29541 tablet_service.cc:1467] Processing CreateTablet for tablet df4355799fe3492884909906d690a634 (DEFAULT_TABLE table=test-table [id=a9725324761d46ffaaa2c6b61a498b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:49.583086 29672 tablet_service.cc:1467] Processing CreateTablet for tablet df4355799fe3492884909906d690a634 (DEFAULT_TABLE table=test-table [id=a9725324761d46ffaaa2c6b61a498b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:49.583086 29410 tablet_service.cc:1467] Processing CreateTablet for tablet df4355799fe3492884909906d690a634 (DEFAULT_TABLE table=test-table [id=a9725324761d46ffaaa2c6b61a498b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:49.583514 29541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4355799fe3492884909906d690a634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:49.584066 29410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4355799fe3492884909906d690a634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:49.584066 29672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4355799fe3492884909906d690a634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:49.589572 29755 tablet_bootstrap.cc:492] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96: Bootstrap starting.
I20260620 20:13:49.590938 29756 tablet_bootstrap.cc:492] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11: Bootstrap starting.
I20260620 20:13:49.592109 29755 tablet_bootstrap.cc:654] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:49.592223 29757 tablet_bootstrap.cc:492] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78: Bootstrap starting.
I20260620 20:13:49.593029 29756 tablet_bootstrap.cc:654] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:49.593006 29755 log.cc:824] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:49.593787 29756 log.cc:824] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:49.594146 29757 tablet_bootstrap.cc:654] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:49.594877 29757 log.cc:824] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:49.595052 29755 tablet_bootstrap.cc:492] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96: No bootstrap required, opened a new log
I20260620 20:13:49.595255 29755 ts_tablet_manager.cc:1397] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:13:49.595678 29756 tablet_bootstrap.cc:492] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11: No bootstrap required, opened a new log
I20260620 20:13:49.595883 29756 ts_tablet_manager.cc:1397] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:13:49.596673 29757 tablet_bootstrap.cc:492] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78: No bootstrap required, opened a new log
I20260620 20:13:49.596865 29757 ts_tablet_manager.cc:1397] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:13:49.599973 29755 raft_consensus.cc:348] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed85c14b4ef040a5b865c38063cffc96" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40095 } } peers { permanent_uuid: "014e29e9a420412e8b539b2779251e78" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 45617 } } peers { permanent_uuid: "022ea45a521542bebed7b91864db5f11" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 42717 } }
I20260620 20:13:49.600087 29756 raft_consensus.cc:348] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed85c14b4ef040a5b865c38063cffc96" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40095 } } peers { permanent_uuid: "014e29e9a420412e8b539b2779251e78" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 45617 } } peers { permanent_uuid: "022ea45a521542bebed7b91864db5f11" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 42717 } }
I20260620 20:13:49.600250 29755 raft_consensus.cc:374] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:49.600271 29756 raft_consensus.cc:374] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:49.600134 29757 raft_consensus.cc:348] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed85c14b4ef040a5b865c38063cffc96" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40095 } } peers { permanent_uuid: "014e29e9a420412e8b539b2779251e78" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 45617 } } peers { permanent_uuid: "022ea45a521542bebed7b91864db5f11" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 42717 } }
I20260620 20:13:49.600329 29756 raft_consensus.cc:729] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022ea45a521542bebed7b91864db5f11, State: Initialized, Role: FOLLOWER
I20260620 20:13:49.600342 29755 raft_consensus.cc:729] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed85c14b4ef040a5b865c38063cffc96, State: Initialized, Role: FOLLOWER
I20260620 20:13:49.600389 29757 raft_consensus.cc:374] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:49.600481 29757 raft_consensus.cc:729] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014e29e9a420412e8b539b2779251e78, State: Initialized, Role: FOLLOWER
I20260620 20:13:49.600829 29756 consensus_queue.cc:260] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed85c14b4ef040a5b865c38063cffc96" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40095 } } peers { permanent_uuid: "014e29e9a420412e8b539b2779251e78" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 45617 } } peers { permanent_uuid: "022ea45a521542bebed7b91864db5f11" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 42717 } }
I20260620 20:13:49.600831 29755 consensus_queue.cc:260] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed85c14b4ef040a5b865c38063cffc96" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40095 } } peers { permanent_uuid: "014e29e9a420412e8b539b2779251e78" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 45617 } } peers { permanent_uuid: "022ea45a521542bebed7b91864db5f11" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 42717 } }
I20260620 20:13:49.600888 29757 consensus_queue.cc:260] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed85c14b4ef040a5b865c38063cffc96" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40095 } } peers { permanent_uuid: "014e29e9a420412e8b539b2779251e78" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 45617 } } peers { permanent_uuid: "022ea45a521542bebed7b91864db5f11" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 42717 } }
I20260620 20:13:49.601495 29738 heartbeater.cc:503] Master 127.24.20.190:43121 was elected leader, sending a full tablet report...
I20260620 20:13:49.601826 29755 ts_tablet_manager.cc:1428] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 20:13:49.601826 29756 ts_tablet_manager.cc:1428] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:13:49.602119 29607 heartbeater.cc:503] Master 127.24.20.190:43121 was elected leader, sending a full tablet report...
I20260620 20:13:49.603433 29757 ts_tablet_manager.cc:1428] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:13:49.603785 29476 heartbeater.cc:503] Master 127.24.20.190:43121 was elected leader, sending a full tablet report...
W20260620 20:13:49.620877 29608 tablet.cc:2369] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:13:49.700917 29477 tablet.cc:2369] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:13:49.706277 29762 raft_consensus.cc:482] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:49.706482 29762 raft_consensus.cc:504] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed85c14b4ef040a5b865c38063cffc96" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40095 } } peers { permanent_uuid: "014e29e9a420412e8b539b2779251e78" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 45617 } } peers { permanent_uuid: "022ea45a521542bebed7b91864db5f11" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 42717 } }
I20260620 20:13:49.707667 29762 leader_election.cc:288] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed85c14b4ef040a5b865c38063cffc96 (127.24.20.130:40095), 014e29e9a420412e8b539b2779251e78 (127.24.20.129:45617)
I20260620 20:13:49.709499 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.707869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:40095 (local address 127.24.20.131:46849)
0620 20:13:49.707996 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:49.708014 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:49.708101 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:49.708357 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:49.708360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:49.708376 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:49.708956 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:13:49.708958 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:49.708969 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:13:49.708994 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:49.709246 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:49.709250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:49.709255 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:13:49.709335 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 20:13:49.709373 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:13:49.709628 29750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.707974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:46849 (local address 127.24.20.130:40095)
0620 20:13:49.708092 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:49.708112 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:49.708131 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:13:49.708184 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:49.708188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:49.708229 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:49.708340 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:49.709119 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:49.709122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:49.709124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:49.709135 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:49.709155 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:49.709163 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:49.709238 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:49.709346 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 20:13:49.709413 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:13:49.709986 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.708158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:45617 (local address 127.24.20.131:42941)
0620 20:13:49.708571 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:49.708583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:49.708714 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:49.708953 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:49.708955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:49.708968 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:49.709447 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:13:49.709449 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:49.709459 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:13:49.709486 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:49.709745 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:49.709747 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:49.709750 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 20:13:49.709830 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 20:13:49.709863 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":122,"threads_started":1}
I20260620 20:13:49.710109 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.708279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:42941 (local address 127.24.20.129:45617)
0620 20:13:49.708452 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:49.708455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:49.708706 (+ 251us) server_negotiation.cc:407] Connection header received
0620 20:13:49.708786 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:49.708790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:49.708829 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:49.708913 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:49.709621 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:49.709625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:49.709628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:49.709640 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:49.709662 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:49.709671 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:49.709760 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:49.709852 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 20:13:49.709890 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:13:49.710286 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4355799fe3492884909906d690a634" candidate_uuid: "022ea45a521542bebed7b91864db5f11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed85c14b4ef040a5b865c38063cffc96" is_pre_election: true
I20260620 20:13:49.710589 29561 raft_consensus.cc:2454] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022ea45a521542bebed7b91864db5f11 in term 0.
I20260620 20:13:49.710659 29430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4355799fe3492884909906d690a634" candidate_uuid: "022ea45a521542bebed7b91864db5f11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014e29e9a420412e8b539b2779251e78" is_pre_election: true
I20260620 20:13:49.710971 29430 raft_consensus.cc:2454] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022ea45a521542bebed7b91864db5f11 in term 0.
I20260620 20:13:49.711015 29626 leader_election.cc:302] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022ea45a521542bebed7b91864db5f11, ed85c14b4ef040a5b865c38063cffc96; no voters:
I20260620 20:13:49.711288 29762 raft_consensus.cc:2781] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:49.711367 29762 raft_consensus.cc:482] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:49.711397 29762 raft_consensus.cc:3037] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:49.712410 29762 raft_consensus.cc:504] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed85c14b4ef040a5b865c38063cffc96" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40095 } } peers { permanent_uuid: "014e29e9a420412e8b539b2779251e78" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 45617 } } peers { permanent_uuid: "022ea45a521542bebed7b91864db5f11" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 42717 } }
I20260620 20:13:49.712760 29762 leader_election.cc:288] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [CANDIDATE]: Term 1 election: Requested vote from peers ed85c14b4ef040a5b865c38063cffc96 (127.24.20.130:40095), 014e29e9a420412e8b539b2779251e78 (127.24.20.129:45617)
I20260620 20:13:49.713124 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4355799fe3492884909906d690a634" candidate_uuid: "022ea45a521542bebed7b91864db5f11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed85c14b4ef040a5b865c38063cffc96"
I20260620 20:13:49.713160 29430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df4355799fe3492884909906d690a634" candidate_uuid: "022ea45a521542bebed7b91864db5f11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "014e29e9a420412e8b539b2779251e78"
I20260620 20:13:49.713228 29561 raft_consensus.cc:3037] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:49.713286 29430 raft_consensus.cc:3037] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:49.714221 29561 raft_consensus.cc:2454] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022ea45a521542bebed7b91864db5f11 in term 1.
I20260620 20:13:49.714375 29430 raft_consensus.cc:2454] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022ea45a521542bebed7b91864db5f11 in term 1.
I20260620 20:13:49.714464 29626 leader_election.cc:302] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022ea45a521542bebed7b91864db5f11, ed85c14b4ef040a5b865c38063cffc96; no voters:
I20260620 20:13:49.714607 29762 raft_consensus.cc:2781] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:49.714802 29762 raft_consensus.cc:686] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [term 1 LEADER]: Becoming Leader. State: Replica: 022ea45a521542bebed7b91864db5f11, State: Running, Role: LEADER
I20260620 20:13:49.715135 29762 consensus_queue.cc:237] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed85c14b4ef040a5b865c38063cffc96" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40095 } } peers { permanent_uuid: "014e29e9a420412e8b539b2779251e78" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 45617 } } peers { permanent_uuid: "022ea45a521542bebed7b91864db5f11" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 42717 } }
I20260620 20:13:49.718077 29287 catalog_manager.cc:5495] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 022ea45a521542bebed7b91864db5f11 (127.24.20.131). New cstate: current_term: 1 leader_uuid: "022ea45a521542bebed7b91864db5f11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed85c14b4ef040a5b865c38063cffc96" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014e29e9a420412e8b539b2779251e78" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "022ea45a521542bebed7b91864db5f11" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 42717 } health_report { overall_health: HEALTHY } } }
I20260620 20:13:49.730218 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.728731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.24.20.131:42717)
0620 20:13:49.728857 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:49.728863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:49.728878 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:13:49.729073 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:49.729076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:49.729130 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:49.729219 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:49.729823 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:49.729827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:49.729829 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:49.729840 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:49.729859 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:49.729868 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:49.729970 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:49.730058 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 20:13:49.730098 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:13:49.735683 29561 raft_consensus.cc:1261] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96 [term 1 FOLLOWER]: Refusing update from remote peer 022ea45a521542bebed7b91864db5f11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:49.735730 29430 raft_consensus.cc:1261] T df4355799fe3492884909906d690a634 P 014e29e9a420412e8b539b2779251e78 [term 1 FOLLOWER]: Refusing update from remote peer 022ea45a521542bebed7b91864db5f11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:49.736367 29765 consensus_queue.cc:1035] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed85c14b4ef040a5b865c38063cffc96" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 40095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:49.736640 29762 consensus_queue.cc:1035] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "014e29e9a420412e8b539b2779251e78" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:49.743366 29768 mvcc.cc:205] Tried to move back new op lower bound from 7299016416190394368 to 7299016416115666944. Current Snapshot: MvccSnapshot[applied={T|T < 7299016416190394368}]
I20260620 20:13:49.744479 29770 mvcc.cc:205] Tried to move back new op lower bound from 7299016416190394368 to 7299016416115666944. Current Snapshot: MvccSnapshot[applied={T|T < 7299016416190394368}]
I20260620 20:13:49.747680 29769 mvcc.cc:205] Tried to move back new op lower bound from 7299016416190394368 to 7299016416115666944. Current Snapshot: MvccSnapshot[applied={T|T < 7299016416190394368}]
I20260620 20:13:49.752279 29287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:49.752440 29287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:49.754752 29287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cc09970e78c4533b017e29b2255c43e: Sending DeleteTablet for 3 replicas of tablet df4355799fe3492884909906d690a634
I20260620 20:13:49.755515 29410 tablet_service.cc:1514] Processing DeleteTablet for tablet df4355799fe3492884909906d690a634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:49 UTC) from {username='slave'} at 127.0.0.1:59502
I20260620 20:13:49.755545 29541 tablet_service.cc:1514] Processing DeleteTablet for tablet df4355799fe3492884909906d690a634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:49 UTC) from {username='slave'} at 127.0.0.1:58098
I20260620 20:13:49.755957 29778 tablet_replica.cc:330] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96: stopping tablet replica
I20260620 20:13:49.755982 29672 tablet_service.cc:1514] Processing DeleteTablet for tablet df4355799fe3492884909906d690a634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:49 UTC) from {username='slave'} at 127.0.0.1:36460
I20260620 20:13:49.756275 29778 raft_consensus.cc:2229] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:49.756387 29780 tablet_replica.cc:330] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11: stopping tablet replica
I20260620 20:13:49.756659 29778 raft_consensus.cc:2258] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:49.756723 29780 raft_consensus.cc:2229] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:13:49.756927 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 29348
I20260620 20:13:49.759397 29778 ts_tablet_manager.cc:1905] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:49.759570 29780 raft_consensus.cc:2258] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:49.761111 29780 ts_tablet_manager.cc:1905] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:49.762495 29778 ts_tablet_manager.cc:1918] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:49.762640 29778 log.cc:1196] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/df4355799fe3492884909906d690a634
I20260620 20:13:49.763000 29778 ts_tablet_manager.cc:1939] T df4355799fe3492884909906d690a634 P ed85c14b4ef040a5b865c38063cffc96: Deleting consensus metadata
W20260620 20:13:49.764063 29273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:45617 (error 108)
I20260620 20:13:49.764402 29780 ts_tablet_manager.cc:1918] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:49.764483 29780 log.cc:1196] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/df4355799fe3492884909906d690a634
I20260620 20:13:49.764535 29273 catalog_manager.cc:4841] TS ed85c14b4ef040a5b865c38063cffc96 (127.24.20.130:40095): tablet df4355799fe3492884909906d690a634 (table test-table [id=a9725324761d46ffaaa2c6b61a498b2f]) successfully deleted
I20260620 20:13:49.764837 29780 ts_tablet_manager.cc:1939] T df4355799fe3492884909906d690a634 P 022ea45a521542bebed7b91864db5f11: Deleting consensus metadata
I20260620 20:13:49.765096 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 29479
I20260620 20:13:49.768028 29275 catalog_manager.cc:4841] TS 022ea45a521542bebed7b91864db5f11 (127.24.20.131:42717): tablet df4355799fe3492884909906d690a634 (table test-table [id=a9725324761d46ffaaa2c6b61a498b2f]) successfully deleted
I20260620 20:13:49.768239 29751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.766976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:45617 (local address 127.0.0.1:59512)
0620 20:13:49.768083 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:49.768154 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:45617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1009}
W20260620 20:13:49.768668 29273 catalog_manager.cc:4584] TS 014e29e9a420412e8b539b2779251e78 (127.24.20.129:45617): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df4355799fe3492884909906d690a634: Network error: Client connection negotiation failed: client connection to 127.24.20.129:45617: connect: Connection refused (error 111)
I20260620 20:13:49.770300 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 29610
I20260620 20:13:49.775890 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 29257
2026-06-20T20:13:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (982 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 20:13:49.797500 24658 mini_oidc.cc:152] Starting JWKS server
I20260620 20:13:49.799100 24658 webserver.cc:460] Webserver started at https://127.0.0.1:36559/ using document root <none> and password file <none>
I20260620 20:13:49.799268 24658 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:13:49.799549 24658 webserver.cc:460] Webserver started at http://127.0.0.1:35559/ using document root <none> and password file <none>
2026-06-20T20:13:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:49Z Disabled control of system clock
I20260620 20:13:49.808501 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:39583
--webserver_interface=127.24.20.190
--webserver_port=0
--builtin_ntp_servers=127.24.20.148:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:39583
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36559/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/testchainca.cert with env {}
W20260620 20:13:49.914443 29787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:49.914677 29787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:49.914714 29787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:49.918629 29787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:49.918677 29787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:49.918692 29787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:49.918708 29787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:49.918764 29787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:13:49.918793 29787 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:13:49.918821 29787 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36559/jwks.json
I20260620 20:13:49.924064 29787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:39583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:39583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36559/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:49.925027 29787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:49.925964 29787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:49.933959 29793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:49.934113 29787 server_base.cc:1034] running on GCE node
W20260620 20:13:49.933964 29792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:49.933995 29795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:49.934801 29787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:49.935853 29787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:49.937041 29787 hybrid_clock.cc:648] HybridClock initialized: now 1781986429937023 us; error 32 us; skew 500 ppm
I20260620 20:13:49.938742 29787 webserver.cc:460] Webserver started at http://127.24.20.190:35161/ using document root <none> and password file <none>
I20260620 20:13:49.939301 29787 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:49.939388 29787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:49.939605 29787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:49.941589 29787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "54cc7e71a0ac440ea6096b8f274a4a3f"
format_stamp: "Formatted at 2026-06-20 20:13:49 on dist-test-slave-pflr"
server_key: "6d68c4074095ea5e67d3fe7b08f8987d"
server_key_iv: "30f4b57b1eec9558c98a221d41080aff"
server_key_version: "encryptionkey@0"
I20260620 20:13:49.942126 29787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "54cc7e71a0ac440ea6096b8f274a4a3f"
format_stamp: "Formatted at 2026-06-20 20:13:49 on dist-test-slave-pflr"
server_key: "6d68c4074095ea5e67d3fe7b08f8987d"
server_key_iv: "30f4b57b1eec9558c98a221d41080aff"
server_key_version: "encryptionkey@0"
I20260620 20:13:49.945580 29787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:13:49.947933 29801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:49.949189 29787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:49.949334 29787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "54cc7e71a0ac440ea6096b8f274a4a3f"
format_stamp: "Formatted at 2026-06-20 20:13:49 on dist-test-slave-pflr"
server_key: "6d68c4074095ea5e67d3fe7b08f8987d"
server_key_iv: "30f4b57b1eec9558c98a221d41080aff"
server_key_version: "encryptionkey@0"
I20260620 20:13:49.949441 29787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:49.976691 29787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:49.977339 29787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:49.977532 29787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:49.983927 29787 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:39583
I20260620 20:13:49.983973 29853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:39583 every 8 connection(s)
I20260620 20:13:49.984869 29787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:49.987867 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:49.993070 29854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f: Bootstrap starting.
I20260620 20:13:49.993995 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 29787
I20260620 20:13:49.994145 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:49.994417 24658 external_mini_cluster.cc:1428] Setting key 4742ee2d6abfc0744df9d45122d2b257
I20260620 20:13:49.995765 29854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:49.996510 29854 log.cc:824] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:49.998528 29854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f: No bootstrap required, opened a new log
I20260620 20:13:50.002756 29854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cc7e71a0ac440ea6096b8f274a4a3f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39583 } }
I20260620 20:13:50.002957 29854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:50.003042 29854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54cc7e71a0ac440ea6096b8f274a4a3f, State: Initialized, Role: FOLLOWER
I20260620 20:13:50.003485 29854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cc7e71a0ac440ea6096b8f274a4a3f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39583 } }
I20260620 20:13:50.003556 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:49.995489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.24.20.190:39583)
0620 20:13:49.996192 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:49.996207 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:49.996241 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:13:49.997027 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:49.997054 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:49.997366 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:49.997639 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:49.997955 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:49.998940 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.000176 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.000766 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.000886 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:50.001712 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:50.001716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:50.001728 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:50.002099 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:50.002152 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:50.002171 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:50.002246 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:50.002406 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 20:13:50.002730 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":149,"threads_started":1}
I20260620 20:13:50.003611 29854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:50.003688 29854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:50.003779 29854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:50.004828 29854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cc7e71a0ac440ea6096b8f274a4a3f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39583 } }
I20260620 20:13:50.005179 29854 leader_election.cc:302] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54cc7e71a0ac440ea6096b8f274a4a3f; no voters:
I20260620 20:13:50.005493 29854 leader_election.cc:288] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:50.005621 29859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:50.006157 29859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [term 1 LEADER]: Becoming Leader. State: Replica: 54cc7e71a0ac440ea6096b8f274a4a3f, State: Running, Role: LEADER
I20260620 20:13:50.006510 29859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cc7e71a0ac440ea6096b8f274a4a3f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39583 } }
I20260620 20:13:50.006721 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:13:50.006708 29854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:50.008003 29860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54cc7e71a0ac440ea6096b8f274a4a3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cc7e71a0ac440ea6096b8f274a4a3f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39583 } } }
I20260620 20:13:50.008144 29860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:50.008559 29861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54cc7e71a0ac440ea6096b8f274a4a3f. Latest consensus state: current_term: 1 leader_uuid: "54cc7e71a0ac440ea6096b8f274a4a3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cc7e71a0ac440ea6096b8f274a4a3f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39583 } } }
I20260620 20:13:50.008733 29861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f [sys.catalog]: This master's current role is: LEADER
W20260620 20:13:50.011593 29874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:13:50.011670 29874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:13:50.011793 29869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:50.012580 29869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:50.018002 29869 catalog_manager.cc:1347] Generated new cluster ID: b1b7c993d44e479a8ce675521832eb9e
I20260620 20:13:50.018064 29869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:50.046329 29869 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:50.047565 29869 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:50.054466 29869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54cc7e71a0ac440ea6096b8f274a4a3f: Generated new TSK 0
I20260620 20:13:50.055160 29869 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 20:13:50.082289 29884 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 20:13:50.082409 29884 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 20:13:50.082899 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:50.066084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51950 (local address 127.24.20.190:39583)
0620 20:13:50.066215 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:50.066220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:50.066410 (+ 190us) server_negotiation.cc:407] Connection header received
0620 20:13:50.066612 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:50.066614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:50.066658 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:50.066739 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:13:50.067013 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.067524 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.068752 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.068972 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.069052 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:50.069396 (+ 344us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:13:50.082583 (+ 13187us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 20:13:50.082796 (+ 213us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51950: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 20:13:50.083002 29857 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51950: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 20:13:50.084317 29883 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:13:50.065997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:39583 (local address 127.0.0.1:51950)
0620 20:13:50.066323 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:50.066339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:50.066462 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:50.066785 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:50.066788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:50.066796 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:13:50.066906 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:50.066913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.067635 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.067638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:50.068647 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:50.068657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.069084 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.069088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:50.069183 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:50.069231 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:13:50.082798 (+ 13567us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 20:13:50.083032 (+ 234us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.20.190:39583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":106,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.24.20.190:39583: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 20:13:50.085382 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 29787
2026-06-20T20:13:50Z chronyd exiting
I20260620 20:13:50.122936 24658 test_util.cc:175] -----------------------------------------------
I20260620 20:13:50.123023 24658 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986405628821-24658-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 20:13:50.123998 24658 mini_oidc.cc:152] Starting JWKS server
I20260620 20:13:50.124893 24658 webserver.cc:460] Webserver started at https://127.0.0.1:46645/ using document root <none> and password file <none>
I20260620 20:13:50.124975 24658 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:13:50.125175 24658 webserver.cc:460] Webserver started at http://127.0.0.1:36463/ using document root <none> and password file <none>
2026-06-20T20:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:50Z Disabled control of system clock
I20260620 20:13:50.133786 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:37091
--webserver_interface=127.24.20.190
--webserver_port=0
--builtin_ntp_servers=127.24.20.148:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:37091
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46645/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/testchainca.cert with env {}
W20260620 20:13:50.240608 29890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:50.240854 29890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:50.240892 29890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:50.244867 29890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:50.244921 29890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:50.244936 29890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:50.244952 29890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:50.244967 29890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:13:50.244983 29890 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:13:50.244999 29890 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46645/jwks.json
I20260620 20:13:50.250058 29890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:37091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:37091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46645/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:50.250955 29890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:50.251883 29890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:50.260044 29896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:50.260085 29898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:50.260056 29895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:50.260313 29890 server_base.cc:1034] running on GCE node
I20260620 20:13:50.260998 29890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:50.262015 29890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:50.263201 29890 hybrid_clock.cc:648] HybridClock initialized: now 1781986430263186 us; error 27 us; skew 500 ppm
I20260620 20:13:50.264990 29890 webserver.cc:460] Webserver started at http://127.24.20.190:37043/ using document root <none> and password file <none>
I20260620 20:13:50.265528 29890 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:50.265614 29890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:50.265831 29890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:50.267711 29890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "56279051fee04ff683ffc56144889430"
format_stamp: "Formatted at 2026-06-20 20:13:50 on dist-test-slave-pflr"
server_key: "8b34f2c59f8a3a154d91354b3f7a64c9"
server_key_iv: "8c41c2f3e9c2a91d4cb8260ffd2108b6"
server_key_version: "encryptionkey@0"
I20260620 20:13:50.268245 29890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "56279051fee04ff683ffc56144889430"
format_stamp: "Formatted at 2026-06-20 20:13:50 on dist-test-slave-pflr"
server_key: "8b34f2c59f8a3a154d91354b3f7a64c9"
server_key_iv: "8c41c2f3e9c2a91d4cb8260ffd2108b6"
server_key_version: "encryptionkey@0"
I20260620 20:13:50.271616 29890 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:13:50.273839 29904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:50.274968 29890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:50.275098 29890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "56279051fee04ff683ffc56144889430"
format_stamp: "Formatted at 2026-06-20 20:13:50 on dist-test-slave-pflr"
server_key: "8b34f2c59f8a3a154d91354b3f7a64c9"
server_key_iv: "8c41c2f3e9c2a91d4cb8260ffd2108b6"
server_key_version: "encryptionkey@0"
I20260620 20:13:50.275203 29890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:50.291356 29890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:50.291934 29890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:50.292115 29890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:50.299628 29890 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:37091
I20260620 20:13:50.299639 29956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:37091 every 8 connection(s)
I20260620 20:13:50.300688 29890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:50.304028 29957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:50.309213 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 29890
I20260620 20:13:50.309350 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:50.309669 24658 external_mini_cluster.cc:1428] Setting key a11ed8efb5a0103f67bb1f6115504ee3
I20260620 20:13:50.309995 29957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430: Bootstrap starting.
I20260620 20:13:50.312564 29957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:50.313392 29957 log.cc:824] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:50.315274 29957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430: No bootstrap required, opened a new log
I20260620 20:13:50.319692 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:50.311085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.24.20.190:37091)
0620 20:13:50.311767 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:50.311781 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:50.311816 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:13:50.312484 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:50.312500 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:50.312953 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:50.313306 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:50.313734 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.314755 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.315793 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.316292 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.316481 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:50.317420 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:50.317428 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:50.317444 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:50.317960 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:50.318051 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:50.318087 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:50.318214 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:50.318376 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 20:13:50.318719 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":145,"threads_started":1}
I20260620 20:13:50.319576 29957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56279051fee04ff683ffc56144889430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37091 } }
I20260620 20:13:50.319828 29957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:50.319885 29957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56279051fee04ff683ffc56144889430, State: Initialized, Role: FOLLOWER
I20260620 20:13:50.320365 29957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56279051fee04ff683ffc56144889430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37091 } }
I20260620 20:13:50.320533 29957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:50.320640 29957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:50.321426 29957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:50.322528 29957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56279051fee04ff683ffc56144889430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37091 } }
I20260620 20:13:50.322587 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:13:50.322882 29957 leader_election.cc:302] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56279051fee04ff683ffc56144889430; no voters:
I20260620 20:13:50.323208 29957 leader_election.cc:288] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:50.323283 29962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:50.323496 29962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [term 1 LEADER]: Becoming Leader. State: Replica: 56279051fee04ff683ffc56144889430, State: Running, Role: LEADER
I20260620 20:13:50.323808 29962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56279051fee04ff683ffc56144889430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37091 } }
I20260620 20:13:50.324514 29957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:50.325033 29964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56279051fee04ff683ffc56144889430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56279051fee04ff683ffc56144889430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37091 } } }
I20260620 20:13:50.325054 29962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56279051fee04ff683ffc56144889430. Latest consensus state: current_term: 1 leader_uuid: "56279051fee04ff683ffc56144889430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56279051fee04ff683ffc56144889430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 37091 } } }
I20260620 20:13:50.325392 29962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:50.325280 29964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:50.325896 29969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:50.328729 29969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:50.333849 29969 catalog_manager.cc:1347] Generated new cluster ID: f8d1e5e7c3b1406caa321c0bea41ec60
I20260620 20:13:50.333914 29969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:50.342613 29969 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:50.343468 29969 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:50.352954 29969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56279051fee04ff683ffc56144889430: Generated new TSK 0
I20260620 20:13:50.353673 29969 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 20:13:50.385699 29987 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 20:13:50.385811 29987 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 20:13:50.386317 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:50.370488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.24.20.190:37091)
0620 20:13:50.370628 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:50.370632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:50.370769 (+ 137us) server_negotiation.cc:407] Connection header received
0620 20:13:50.370906 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:50.370909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:50.370966 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:50.371074 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:13:50.371327 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.372150 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.373052 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.373419 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.373523 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:50.373834 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:13:50.385939 (+ 12105us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 20:13:50.386184 (+ 245us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60200: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 20:13:50.386379 29960 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60200: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 20:13:50.386427 29986 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:13:50.370356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:37091 (local address 127.0.0.1:60200)
0620 20:13:50.370691 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:50.370704 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:50.370807 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:50.371104 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:50.371107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:50.371115 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:13:50.371218 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:50.371223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.372262 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.372264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:50.372949 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:50.372957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.373571 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.373574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:50.373649 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:50.373670 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:13:50.386130 (+ 12460us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 20:13:50.386258 (+ 128us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.20.190:37091: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":126,"threads_started":1}
I20260620 20:13:50.387380 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 29890
2026-06-20T20:13:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 20:13:50.426505 24658 mini_oidc.cc:152] Starting JWKS server
I20260620 20:13:50.427376 24658 webserver.cc:460] Webserver started at https://127.0.0.1:38327/ using document root <none> and password file <none>
I20260620 20:13:50.427491 24658 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:13:50.427757 24658 webserver.cc:460] Webserver started at http://127.0.0.1:34075/ using document root <none> and password file <none>
2026-06-20T20:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:50Z Disabled control of system clock
I20260620 20:13:50.436035 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:41449
--webserver_interface=127.24.20.190
--webserver_port=0
--builtin_ntp_servers=127.24.20.148:34559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:41449
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38327/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 20:13:50.541589 29993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:50.541889 29993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:50.541956 29993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:50.545912 29993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:50.546005 29993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:50.546034 29993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:50.546069 29993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:50.546105 29993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:13:50.546144 29993 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:13:50.546175 29993 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38327/jwks.json
I20260620 20:13:50.550993 29993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:34559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:41449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:41449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38327/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:50.551997 29993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:50.553330 29993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:50.561466 30001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:50.561478 29998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:50.561519 29999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:50.561612 29993 server_base.cc:1034] running on GCE node
I20260620 20:13:50.562235 29993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:50.563292 29993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:50.564448 29993 hybrid_clock.cc:648] HybridClock initialized: now 1781986430564416 us; error 44 us; skew 500 ppm
I20260620 20:13:50.566293 29993 webserver.cc:460] Webserver started at http://127.24.20.190:33987/ using document root <none> and password file <none>
I20260620 20:13:50.566836 29993 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:50.566897 29993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:50.567139 29993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:50.569139 29993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "466b172760ea4e0095a27f6ed14ba530"
format_stamp: "Formatted at 2026-06-20 20:13:50 on dist-test-slave-pflr"
server_key: "41f3c485830a1a11df7ce111d2122720"
server_key_iv: "f64defb0559b85b0022ffa817c7a67d2"
server_key_version: "encryptionkey@0"
I20260620 20:13:50.569674 29993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "466b172760ea4e0095a27f6ed14ba530"
format_stamp: "Formatted at 2026-06-20 20:13:50 on dist-test-slave-pflr"
server_key: "41f3c485830a1a11df7ce111d2122720"
server_key_iv: "f64defb0559b85b0022ffa817c7a67d2"
server_key_version: "encryptionkey@0"
I20260620 20:13:50.573192 29993 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:13:50.575419 30007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:50.576566 29993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:50.576754 29993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "466b172760ea4e0095a27f6ed14ba530"
format_stamp: "Formatted at 2026-06-20 20:13:50 on dist-test-slave-pflr"
server_key: "41f3c485830a1a11df7ce111d2122720"
server_key_iv: "f64defb0559b85b0022ffa817c7a67d2"
server_key_version: "encryptionkey@0"
I20260620 20:13:50.576864 29993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:50.596117 29993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:50.596812 29993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:50.597000 29993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:50.603628 29993 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:41449
I20260620 20:13:50.603657 30059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:41449 every 8 connection(s)
I20260620 20:13:50.604494 29993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:50.607398 30060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:50.611138 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 29993
I20260620 20:13:50.611264 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:50.611570 24658 external_mini_cluster.cc:1428] Setting key 6bd9eeafa920303bf556cb3bf8380d0a
I20260620 20:13:50.613152 30060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530: Bootstrap starting.
I20260620 20:13:50.615200 30060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:50.615893 30060 log.cc:824] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:50.617652 30060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530: No bootstrap required, opened a new log
I20260620 20:13:50.620482 30060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466b172760ea4e0095a27f6ed14ba530" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41449 } }
I20260620 20:13:50.620707 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:50.612898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.24.20.190:41449)
0620 20:13:50.613505 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:50.613520 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:50.613554 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:13:50.614150 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:50.614165 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:50.614530 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:50.614852 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:50.615289 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.616316 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.617249 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.617727 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.617901 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:50.618731 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:50.618743 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:50.618755 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:50.619109 (+ 354us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:50.619168 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:50.619186 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:50.619290 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:50.619399 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 20:13:50.619633 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
I20260620 20:13:50.620744 30060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:50.620816 30060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 466b172760ea4e0095a27f6ed14ba530, State: Initialized, Role: FOLLOWER
I20260620 20:13:50.621261 30060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466b172760ea4e0095a27f6ed14ba530" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41449 } }
I20260620 20:13:50.621380 30060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:50.621461 30060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:50.621584 30060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:50.622617 30060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466b172760ea4e0095a27f6ed14ba530" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41449 } }
I20260620 20:13:50.622969 30060 leader_election.cc:302] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 466b172760ea4e0095a27f6ed14ba530; no voters:
I20260620 20:13:50.623303 30060 leader_election.cc:288] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:50.623385 30065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:50.623512 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:13:50.623575 30065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [term 1 LEADER]: Becoming Leader. State: Replica: 466b172760ea4e0095a27f6ed14ba530, State: Running, Role: LEADER
I20260620 20:13:50.623850 30065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466b172760ea4e0095a27f6ed14ba530" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41449 } }
I20260620 20:13:50.624330 30060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:50.624971 30067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 466b172760ea4e0095a27f6ed14ba530. Latest consensus state: current_term: 1 leader_uuid: "466b172760ea4e0095a27f6ed14ba530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466b172760ea4e0095a27f6ed14ba530" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41449 } } }
I20260620 20:13:50.625144 30067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:50.625399 30066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "466b172760ea4e0095a27f6ed14ba530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466b172760ea4e0095a27f6ed14ba530" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41449 } } }
I20260620 20:13:50.625561 30066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:50.625761 30073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:50.628760 30073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:50.633498 30073 catalog_manager.cc:1347] Generated new cluster ID: a498731a0df34ee287955fcf56e46350
I20260620 20:13:50.633571 30073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:50.647492 30073 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:50.648428 30073 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:50.652369 30073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 466b172760ea4e0095a27f6ed14ba530: Generated new TSK 0
I20260620 20:13:50.653090 30073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:13:50.672173 30063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:50.667286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.24.20.190:41449)
0620 20:13:50.667414 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:50.667418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:50.667554 (+ 136us) server_negotiation.cc:407] Connection header received
0620 20:13:50.667685 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:50.667688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:50.667741 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:50.667849 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:13:50.668057 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.668933 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.669765 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.670112 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.670220 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:50.670472 (+ 252us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:13:50.671822 (+ 1350us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38327/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 20:13:50.672036 (+ 214us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56930: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38327/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 20:13:50.672241 30063 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56930: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38327/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 20:13:50.673919 30089 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:13:50.667162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:41449 (local address 127.0.0.1:56930)
0620 20:13:50.667473 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:50.667486 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:50.667583 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:50.667865 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:50.667867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:50.667873 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:13:50.667963 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:50.667967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.669037 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.669040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:50.669653 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:50.669660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.670231 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.670233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:50.670299 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:50.670319 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:13:50.671979 (+ 1660us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38327/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 20:13:50.673758 (+ 1779us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.20.190:41449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38327/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":1604,"thread_start_us":107,"threads_started":1}
I20260620 20:13:50.674562 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 29993
E20260620 20:13:50.685217 30090 webserver.cc:533] Webserver: sslize failed:
E20260620 20:13:50.685276 30090 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T20:13:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (287 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T20:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:50Z Disabled control of system clock
I20260620 20:13:50.721932 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:42061
--webserver_interface=127.24.20.190
--webserver_port=0
--builtin_ntp_servers=127.24.20.148:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:42061
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/testchainca.cert with env {}
W20260620 20:13:50.827713 30094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:50.827956 30094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:50.827996 30094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:50.831991 30094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:50.832044 30094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:50.832059 30094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:50.832077 30094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:50.832091 30094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:13:50.832150 30094 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:13:50.832170 30094 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 20:13:50.837456 30094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:42061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:42061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:50.838425 30094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:50.839378 30094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:50.847398 30102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:50.847412 30100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:50.847407 30099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:50.847621 30094 server_base.cc:1034] running on GCE node
I20260620 20:13:50.848232 30094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:50.849277 30094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:50.850454 30094 hybrid_clock.cc:648] HybridClock initialized: now 1781986430850425 us; error 34 us; skew 500 ppm
I20260620 20:13:50.852304 30094 webserver.cc:460] Webserver started at http://127.24.20.190:43659/ using document root <none> and password file <none>
I20260620 20:13:50.852928 30094 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:50.852993 30094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:50.853221 30094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:50.855120 30094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "0af7372af9c84e1a883c40bebf442b69"
format_stamp: "Formatted at 2026-06-20 20:13:50 on dist-test-slave-pflr"
server_key: "dfc6a0f2dfe1982059987fb251c20136"
server_key_iv: "f73c2eaa07fc6b1df8b2437e2cc1a6c0"
server_key_version: "encryptionkey@0"
I20260620 20:13:50.855664 30094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "0af7372af9c84e1a883c40bebf442b69"
format_stamp: "Formatted at 2026-06-20 20:13:50 on dist-test-slave-pflr"
server_key: "dfc6a0f2dfe1982059987fb251c20136"
server_key_iv: "f73c2eaa07fc6b1df8b2437e2cc1a6c0"
server_key_version: "encryptionkey@0"
I20260620 20:13:50.859140 30094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 20:13:50.861524 30108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:50.862784 30094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:50.862900 30094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "0af7372af9c84e1a883c40bebf442b69"
format_stamp: "Formatted at 2026-06-20 20:13:50 on dist-test-slave-pflr"
server_key: "dfc6a0f2dfe1982059987fb251c20136"
server_key_iv: "f73c2eaa07fc6b1df8b2437e2cc1a6c0"
server_key_version: "encryptionkey@0"
I20260620 20:13:50.863001 30094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:50.889081 30094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:50.889775 30094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:50.889976 30094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:50.896772 30094 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:42061
I20260620 20:13:50.896785 30160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:42061 every 8 connection(s)
I20260620 20:13:50.897747 30094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:50.900727 30161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:50.906665 30161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69: Bootstrap starting.
I20260620 20:13:50.907364 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 30094
I20260620 20:13:50.907500 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:50.907807 24658 external_mini_cluster.cc:1428] Setting key f5ec8ad8f5cbb20a73b255987be82b1c
I20260620 20:13:50.909489 30161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:50.910203 30161 log.cc:824] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:50.911975 30161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69: No bootstrap required, opened a new log
I20260620 20:13:50.915438 30161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af7372af9c84e1a883c40bebf442b69" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42061 } }
I20260620 20:13:50.915740 30161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:50.915819 30161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af7372af9c84e1a883c40bebf442b69, State: Initialized, Role: FOLLOWER
I20260620 20:13:50.916241 30161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af7372af9c84e1a883c40bebf442b69" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42061 } }
I20260620 20:13:50.916388 30161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:50.916481 30161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:50.916576 30161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:50.916570 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:50.908943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54872 (local address 127.24.20.190:42061)
0620 20:13:50.909540 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:50.909554 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:50.909610 (+ 56us) server_negotiation.cc:407] Connection header received
0620 20:13:50.910237 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:50.910253 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:50.910674 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:50.910994 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:50.911400 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.912370 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.913318 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.913779 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.913952 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:50.914777 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:50.914782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:50.914795 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:13:50.915160 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:50.915211 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:13:50.915230 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:50.915336 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:50.915445 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 20:13:50.915682 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":127,"threads_started":1}
I20260620 20:13:50.917675 30161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af7372af9c84e1a883c40bebf442b69" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42061 } }
I20260620 20:13:50.918030 30161 leader_election.cc:302] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0af7372af9c84e1a883c40bebf442b69; no voters:
I20260620 20:13:50.918318 30161 leader_election.cc:288] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:50.918416 30166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:50.918678 30166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [term 1 LEADER]: Becoming Leader. State: Replica: 0af7372af9c84e1a883c40bebf442b69, State: Running, Role: LEADER
I20260620 20:13:50.919021 30166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af7372af9c84e1a883c40bebf442b69" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42061 } }
I20260620 20:13:50.919445 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:13:50.919517 30161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:50.920327 30166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0af7372af9c84e1a883c40bebf442b69. Latest consensus state: current_term: 1 leader_uuid: "0af7372af9c84e1a883c40bebf442b69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af7372af9c84e1a883c40bebf442b69" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42061 } } }
I20260620 20:13:50.920442 30166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:50.920789 30168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0af7372af9c84e1a883c40bebf442b69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af7372af9c84e1a883c40bebf442b69" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42061 } } }
I20260620 20:13:50.920965 30168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:50.921072 30175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:50.924037 30175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:50.929267 30175 catalog_manager.cc:1347] Generated new cluster ID: 8a4bb49b747b4de288699bb325dbae35
I20260620 20:13:50.929325 30175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:50.950531 30175 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:50.951808 30175 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:50.955976 30175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0af7372af9c84e1a883c40bebf442b69: Generated new TSK 0
I20260620 20:13:50.956684 30175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:13:51.462354 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:50.968343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.24.20.190:42061)
0620 20:13:50.968484 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:50.968489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:50.968647 (+ 158us) server_negotiation.cc:407] Connection header received
0620 20:13:50.968806 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:50.968808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:50.968853 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:50.968938 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:13:50.969188 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.969696 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.970881 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.971088 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.971169 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:50.971459 (+ 290us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:13:51.461974 (+490515us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 20:13:51.462236 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54882: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 20:13:51.462483 30164 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54882: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 20:13:51.462642 30190 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:13:50.968260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42061 (local address 127.0.0.1:54882)
0620 20:13:50.968532 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:50.968546 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:50.968692 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:50.968959 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:50.968962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:50.968969 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:13:50.969085 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:50.969092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.969796 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.969799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:50.970778 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:50.970787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:50.971201 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:50.971205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:50.971291 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:50.971317 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:13:51.462205 (+490888us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 20:13:51.462402 (+ 197us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.20.190:42061: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":103,"threads_started":1}
I20260620 20:13:51.463248 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 30094
2026-06-20T20:13:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (768 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (118 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:13:51 dist-test-slave-pflr krb5kdc[30202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:13:51 dist-test-slave-pflr krb5kdc[30202](info): set up 2 sockets
Jun 20 20:13:51 dist-test-slave-pflr krb5kdc[30202](info): commencing operation
krb5kdc: starting...
W20260620 20:13:52.749213 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:13:52 dist-test-slave-pflr krb5kdc[30202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:13:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:13:52.868032 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:39803
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:43839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:39803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:52.974319 30215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:52.974563 30215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:52.974601 30215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:52.978565 30215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:52.978616 30215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:52.978636 30215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:52.978652 30215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:52.978696 30215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:52.983784 30215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:39803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:39803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:52.984781 30215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:52.985766 30215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:52.993830 30220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:52.993850 30223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:52.993834 30221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:52.994179 30215 server_base.cc:1034] running on GCE node
I20260620 20:13:52.994827 30215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:52.996177 30215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:52.997350 30215 hybrid_clock.cc:648] HybridClock initialized: now 1781986432997324 us; error 43 us; skew 500 ppm
Jun 20 20:13:52 dist-test-slave-pflr krb5kdc[30202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986432, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:53.000803 30215 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:13:53.001796 30215 webserver.cc:460] Webserver started at http://127.24.20.190:38901/ using document root <none> and password file <none>
I20260620 20:13:53.002470 30215 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:53.002532 30215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:53.002784 30215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:53.005301 30215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "09547fbfd6dc474b90441c9f56715073"
format_stamp: "Formatted at 2026-06-20 20:13:52 on dist-test-slave-pflr"
server_key: "5a8e45ba43905c5f22a5ba6f64f87f24"
server_key_iv: "47b22949d663710ec5647d09b7edaa62"
server_key_version: "encryptionkey@0"
I20260620 20:13:53.005782 30215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "09547fbfd6dc474b90441c9f56715073"
format_stamp: "Formatted at 2026-06-20 20:13:52 on dist-test-slave-pflr"
server_key: "5a8e45ba43905c5f22a5ba6f64f87f24"
server_key_iv: "47b22949d663710ec5647d09b7edaa62"
server_key_version: "encryptionkey@0"
I20260620 20:13:53.009301 30215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:13:53.011552 30230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:53.012781 30215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 20:13:53.012902 30215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "09547fbfd6dc474b90441c9f56715073"
format_stamp: "Formatted at 2026-06-20 20:13:52 on dist-test-slave-pflr"
server_key: "5a8e45ba43905c5f22a5ba6f64f87f24"
server_key_iv: "47b22949d663710ec5647d09b7edaa62"
server_key_version: "encryptionkey@0"
I20260620 20:13:53.013010 30215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:53.033170 30215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:53.035982 30215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:53.036185 30215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:53.043043 30215 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:39803
I20260620 20:13:53.043061 30282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:39803 every 8 connection(s)
I20260620 20:13:53.043998 30215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:53.046604 30283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:53.051791 30283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073: Bootstrap starting.
I20260620 20:13:53.054082 30283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:53.054136 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 30215
I20260620 20:13:53.054260 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:53.054572 24658 external_mini_cluster.cc:1428] Setting key 70a46f9069ba7675088f90454ed2550e
I20260620 20:13:53.054884 30283 log.cc:824] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:53.056819 30283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073: No bootstrap required, opened a new log
I20260620 20:13:53.060943 30283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09547fbfd6dc474b90441c9f56715073" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39803 } }
I20260620 20:13:53.061160 30283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:53.061241 30283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09547fbfd6dc474b90441c9f56715073, State: Initialized, Role: FOLLOWER
I20260620 20:13:53.061719 30283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09547fbfd6dc474b90441c9f56715073" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39803 } }
I20260620 20:13:53.061905 30283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:53.061981 30283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:53.062096 30283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:53.063310 30283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09547fbfd6dc474b90441c9f56715073" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39803 } }
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:53.063699 30283 leader_election.cc:302] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09547fbfd6dc474b90441c9f56715073; no voters:
I20260620 20:13:53.064085 30283 leader_election.cc:288] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:53.064180 30288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:53.064366 30288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [term 1 LEADER]: Becoming Leader. State: Replica: 09547fbfd6dc474b90441c9f56715073, State: Running, Role: LEADER
I20260620 20:13:53.064697 30288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09547fbfd6dc474b90441c9f56715073" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39803 } }
I20260620 20:13:53.065366 30283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:53.066244 30289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09547fbfd6dc474b90441c9f56715073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09547fbfd6dc474b90441c9f56715073" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39803 } } }
I20260620 20:13:53.066378 30289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:53.066677 30290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09547fbfd6dc474b90441c9f56715073. Latest consensus state: current_term: 1 leader_uuid: "09547fbfd6dc474b90441c9f56715073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09547fbfd6dc474b90441c9f56715073" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 39803 } } }
I20260620 20:13:53.066761 30290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:53.069741 30287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:53.055643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.24.20.190:39803)
0620 20:13:53.055996 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:53.056020 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:53.056054 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:13:53.056715 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:53.056733 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:53.057096 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:53.057364 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:53.058245 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.059207 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.060516 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.061097 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.061305 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:53.064279 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:53.064289 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:53.064309 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:53.064345 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:53.066462 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.066984 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.066993 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.066999 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.067158 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.067503 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.067509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.067510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.068018 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:53.068232 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:53.068515 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 20:13:53.068829 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":99,"threads_started":1}
W20260620 20:13:53.070614 30303 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:13:53.070698 30303 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:13:53.070763 30297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:53.071748 30297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:53.077059 30297 catalog_manager.cc:1347] Generated new cluster ID: bfdfa663eec244b796491cd8720817ae
I20260620 20:13:53.077131 30297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:53.092458 30297 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:53.093353 30297 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:53.097744 30297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09547fbfd6dc474b90441c9f56715073: Generated new TSK 0
I20260620 20:13:53.098407 30297 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:13:53.135898 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:39803
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:43839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:53.248854 30309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:53.249081 30309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:53.249121 30309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:53.252964 30309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:53.253008 30309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:53.253062 30309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:13:53.257944 30309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:39803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:53.258826 30309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:53.259722 30309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:53.269004 30317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:53.269012 30314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:53.269026 30315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:53.269305 30309 server_base.cc:1034] running on GCE node
I20260620 20:13:53.269722 30309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:53.270359 30309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:53.271557 30309 hybrid_clock.cc:648] HybridClock initialized: now 1781986433271533 us; error 34 us; skew 500 ppm
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:53.274150 30309 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:13:53.274957 30309 webserver.cc:460] Webserver started at http://127.24.20.129:44023/ using document root <none> and password file <none>
I20260620 20:13:53.275509 30309 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:53.275597 30309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:53.275815 30309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:53.277783 30309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "ccf637a29c9e4e6f9abe5aa2d2353bab"
format_stamp: "Formatted at 2026-06-20 20:13:53 on dist-test-slave-pflr"
server_key: "1bd8b5f1246dc4ffdbb935bf4bf99d6f"
server_key_iv: "a664374bf1e333b14d006bc2652be7b0"
server_key_version: "encryptionkey@0"
I20260620 20:13:53.278323 30309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "ccf637a29c9e4e6f9abe5aa2d2353bab"
format_stamp: "Formatted at 2026-06-20 20:13:53 on dist-test-slave-pflr"
server_key: "1bd8b5f1246dc4ffdbb935bf4bf99d6f"
server_key_iv: "a664374bf1e333b14d006bc2652be7b0"
server_key_version: "encryptionkey@0"
I20260620 20:13:53.281826 30309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:13:53.284061 30324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:53.285249 30309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:53.285385 30309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "ccf637a29c9e4e6f9abe5aa2d2353bab"
format_stamp: "Formatted at 2026-06-20 20:13:53 on dist-test-slave-pflr"
server_key: "1bd8b5f1246dc4ffdbb935bf4bf99d6f"
server_key_iv: "a664374bf1e333b14d006bc2652be7b0"
server_key_version: "encryptionkey@0"
I20260620 20:13:53.285491 30309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:53.299135 30309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:53.302318 30309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:53.302546 30309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:53.303108 30309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:53.304121 30309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:53.304185 30309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:53.304251 30309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:53.304312 30309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:53.313503 30309 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:46619
I20260620 20:13:53.313527 30437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:46619 every 8 connection(s)
I20260620 20:13:53.314375 30309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:53.322180 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 30309
I20260620 20:13:53.322325 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:13:53.322674 24658 external_mini_cluster.cc:1428] Setting key 31f29fdb0e47eed5f1931f9561d3b745
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:53.328428 30287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:53.316065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:51611 (local address 127.24.20.190:39803)
0620 20:13:53.316194 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:53.316205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:53.317109 (+ 904us) server_negotiation.cc:407] Connection header received
0620 20:13:53.318474 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:53.318478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:53.318535 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:53.318613 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:53.319698 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.320239 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.321357 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.321587 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.321708 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:53.324659 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:53.324664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:53.324667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:53.324703 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:53.326319 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.326872 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.326878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.326880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.326974 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.327337 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.327342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.327344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.327531 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:53.327643 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:53.328120 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 20:13:53.328246 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:13:53.329229 30440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:53.316447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:39803 (local address 127.24.20.129:51611)
0620 20:13:53.316966 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:53.317006 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:53.318267 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:53.318783 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:53.318789 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:53.319109 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:53.319549 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:53.319560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.320362 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.320366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:53.321234 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:53.321242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.321740 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.321744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:53.321944 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:53.322674 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:53.322696 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:53.324471 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:53.326472 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:53.326478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:53.326492 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:53.326758 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:53.327085 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:53.327088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:53.327090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:53.327233 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:53.327642 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:53.327649 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:53.327883 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 20:13:53.328076 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 20:13:53.328385 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":178,"threads_started":1}
I20260620 20:13:53.330524 30438 heartbeater.cc:347] Connected to a master server at 127.24.20.190:39803
I20260620 20:13:53.330760 30438 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:53.331207 30438 heartbeater.cc:511] Master 127.24.20.190:39803 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:13:53.332778 30247 ts_manager.cc:194] Registered new tserver with Master: ccf637a29c9e4e6f9abe5aa2d2353bab (127.24.20.129:46619)
I20260620 20:13:53.333920 30247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:51611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:13:53.347656 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:39803
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:43839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:53.453804 30443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:53.454049 30443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:53.454089 30443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:53.457998 30443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:53.458045 30443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:53.458132 30443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:13:53.463210 30443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:39803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:53.464196 30443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:53.465287 30443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:53.474514 30448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:53.474486 30449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:53.474501 30451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:53.474738 30443 server_base.cc:1034] running on GCE node
I20260620 20:13:53.475067 30443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:53.475643 30443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:53.476833 30443 hybrid_clock.cc:648] HybridClock initialized: now 1781986433476808 us; error 38 us; skew 500 ppm
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:53.479523 30443 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:13:53.480393 30443 webserver.cc:460] Webserver started at http://127.24.20.130:45169/ using document root <none> and password file <none>
I20260620 20:13:53.480978 30443 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:53.481045 30443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:53.481273 30443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:53.483331 30443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "013c15acf0b847ed80abbfcd10a3927c"
format_stamp: "Formatted at 2026-06-20 20:13:53 on dist-test-slave-pflr"
server_key: "c6450ed9964f164b793b8af019c0cca1"
server_key_iv: "ddd9df2675ed68cbadf7fb5368c86d95"
server_key_version: "encryptionkey@0"
I20260620 20:13:53.483878 30443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "013c15acf0b847ed80abbfcd10a3927c"
format_stamp: "Formatted at 2026-06-20 20:13:53 on dist-test-slave-pflr"
server_key: "c6450ed9964f164b793b8af019c0cca1"
server_key_iv: "ddd9df2675ed68cbadf7fb5368c86d95"
server_key_version: "encryptionkey@0"
I20260620 20:13:53.487380 30443 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:13:53.489825 30458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:53.491057 30443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:53.491146 30443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "013c15acf0b847ed80abbfcd10a3927c"
format_stamp: "Formatted at 2026-06-20 20:13:53 on dist-test-slave-pflr"
server_key: "c6450ed9964f164b793b8af019c0cca1"
server_key_iv: "ddd9df2675ed68cbadf7fb5368c86d95"
server_key_version: "encryptionkey@0"
I20260620 20:13:53.491278 30443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:53.505596 30443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:53.508500 30443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:53.508744 30443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:53.509305 30443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:53.510128 30443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:53.510174 30443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:53.510237 30443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:53.510269 30443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:53.519402 30443 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:40217
I20260620 20:13:53.519421 30571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:40217 every 8 connection(s)
I20260620 20:13:53.520246 30443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:53.523290 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 30443
I20260620 20:13:53.523420 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:13:53.523689 24658 external_mini_cluster.cc:1428] Setting key ec6f24f3bc653c615311a0da33eae68b
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
I20260620 20:13:53.535351 30287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:53.522028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:33859 (local address 127.24.20.190:39803)
0620 20:13:53.522162 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:53.522166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:53.522994 (+ 828us) server_negotiation.cc:407] Connection header received
0620 20:13:53.524746 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:53.524749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:53.524794 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:53.524882 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:53.526412 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.527091 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.528230 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.528663 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.528784 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:53.531500 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:53.531504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:53.531507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:53.531534 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:53.533297 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.533784 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.533786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.533787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.533856 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.534299 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.534301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.534303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.534463 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:53.534535 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:53.535126 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 20:13:53.535232 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
I20260620 20:13:53.536418 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:53.522426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:39803 (local address 127.24.20.130:33859)
0620 20:13:53.522853 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:53.522892 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:53.524477 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:53.525086 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:53.525094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:53.525578 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:53.526260 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:53.526281 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.527237 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.527240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:53.528106 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:53.528115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.528838 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.528841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:53.529017 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:53.529806 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:53.529824 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:53.531326 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:53.533406 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:53.533410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:53.533420 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:53.533699 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:53.534028 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:53.534035 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:53.534040 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:53.534206 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:53.534603 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:53.534608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:53.534834 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 20:13:53.535067 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 20:13:53.535335 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260620 20:13:53.537609 30572 heartbeater.cc:347] Connected to a master server at 127.24.20.190:39803
I20260620 20:13:53.537855 30572 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:53.538362 30572 heartbeater.cc:511] Master 127.24.20.190:39803 requested a full tablet report, sending...
I20260620 20:13:53.539402 30247 ts_manager.cc:194] Registered new tserver with Master: 013c15acf0b847ed80abbfcd10a3927c (127.24.20.130:40217)
I20260620 20:13:53.540246 30247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:33859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:13:53.551433 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:39803
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:43839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:53.658545 30577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:53.658768 30577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:53.658844 30577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:53.662828 30577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:53.662881 30577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:53.662983 30577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:13:53.668318 30577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:39803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:53.669288 30577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:53.670243 30577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:13:53.679550 30577 server_base.cc:1034] running on GCE node
W20260620 20:13:53.679461 30585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:53.679459 30583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:53.679525 30582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:53.680151 30577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:53.680756 30577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:53.681936 30577 hybrid_clock.cc:648] HybridClock initialized: now 1781986433681923 us; error 23 us; skew 500 ppm
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:53.684438 30577 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:13:53.685307 30577 webserver.cc:460] Webserver started at http://127.24.20.131:41471/ using document root <none> and password file <none>
I20260620 20:13:53.685813 30577 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:53.685900 30577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:53.686128 30577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:53.688037 30577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "dd5e0d12ebc141648b222b99f6d83103"
format_stamp: "Formatted at 2026-06-20 20:13:53 on dist-test-slave-pflr"
server_key: "0c329860048f722934f95dbf1c321df6"
server_key_iv: "fa21fde9856c55faf4063ea31c30da64"
server_key_version: "encryptionkey@0"
I20260620 20:13:53.688573 30577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "dd5e0d12ebc141648b222b99f6d83103"
format_stamp: "Formatted at 2026-06-20 20:13:53 on dist-test-slave-pflr"
server_key: "0c329860048f722934f95dbf1c321df6"
server_key_iv: "fa21fde9856c55faf4063ea31c30da64"
server_key_version: "encryptionkey@0"
I20260620 20:13:53.692222 30577 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:13:53.694584 30592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:53.695848 30577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:53.695984 30577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "dd5e0d12ebc141648b222b99f6d83103"
format_stamp: "Formatted at 2026-06-20 20:13:53 on dist-test-slave-pflr"
server_key: "0c329860048f722934f95dbf1c321df6"
server_key_iv: "fa21fde9856c55faf4063ea31c30da64"
server_key_version: "encryptionkey@0"
I20260620 20:13:53.696089 30577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:53.725739 30577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:53.728971 30577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:53.729183 30577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:53.729741 30577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:53.730525 30577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:53.730569 30577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:53.730631 30577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:53.730669 30577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:53.739189 30577 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:35901
I20260620 20:13:53.739217 30705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:35901 every 8 connection(s)
I20260620 20:13:53.740013 30577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:13:53.747184 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 30577
I20260620 20:13:53.747315 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:13:53.747666 24658 external_mini_cluster.cc:1428] Setting key 2618b24a2ea558031ed37795361837dc
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:13:53.753396 30287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:53.741834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:60809 (local address 127.24.20.190:39803)
0620 20:13:53.741982 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:53.741986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:53.742779 (+ 793us) server_negotiation.cc:407] Connection header received
0620 20:13:53.744155 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:53.744159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:53.744219 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:53.744316 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:53.745468 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.745988 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.747090 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.747346 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.747468 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:53.750399 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:53.750402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:53.750404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:53.750426 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:53.751648 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.752101 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.752103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.752104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.752177 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.752460 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.752461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.752462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.752669 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:53.752753 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:53.753186 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 20:13:53.753276 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:13:53.754254 30708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:53.742179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:39803 (local address 127.24.20.131:60809)
0620 20:13:53.742638 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:53.742679 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:53.743936 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:53.744484 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:53.744491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:53.744881 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:53.745330 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:53.745343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.746110 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.746120 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:53.746965 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:53.746974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.747575 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.747582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:53.747801 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:53.748491 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:53.748514 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:53.750251 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:53.751750 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:53.751755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:53.751766 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:53.752017 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:53.752254 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:53.752256 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:53.752257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:53.752384 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:53.752754 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:53.752759 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:53.752980 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 20:13:53.753139 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 20:13:53.753389 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":112,"threads_started":1}
I20260620 20:13:53.755255 30706 heartbeater.cc:347] Connected to a master server at 127.24.20.190:39803
I20260620 20:13:53.755450 30706 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:53.755867 30706 heartbeater.cc:511] Master 127.24.20.190:39803 requested a full tablet report, sending...
I20260620 20:13:53.756848 30247 ts_manager.cc:194] Registered new tserver with Master: dd5e0d12ebc141648b222b99f6d83103 (127.24.20.131:35901)
I20260620 20:13:53.757395 30247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:60809
I20260620 20:13:53.761399 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/test-admin.keytab.
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:53.778196 24658 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 20:13:53.781175 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:13:53.779503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.24.20.129:46619)
0620 20:13:53.779844 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:53.779853 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:53.779877 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:13:53.779923 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:53.779928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:53.780105 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:53.780230 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:53.780927 (+ 697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43056: BlockingRecv error: recv got EOF from 127.0.0.1:43056 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":149,"threads_started":1}
W20260620 20:13:53.781208 24658 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/krb5cc' of type FILE
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:53.782213 24658 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:53.783257 24658 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.129@KRBTEST.COM
I20260620 20:13:53.792833 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:53.784141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.24.20.129:46619)
0620 20:13:53.784241 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:53.784244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:53.784424 (+ 180us) server_negotiation.cc:407] Connection header received
0620 20:13:53.784496 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:53.784498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:53.784541 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:53.784655 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:53.785125 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.785878 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.786648 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.786995 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.787093 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:53.789409 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:53.789418 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:53.789430 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:53.789451 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:53.791065 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.791526 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.791539 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.791540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.791612 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.791911 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.791914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.791916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.792142 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:53.792343 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:53.792523 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 20:13:53.792686 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 20:13:53.796415 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:13:53.795628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.24.20.129:46619)
0620 20:13:53.795740 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:53.795744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:53.795809 (+ 65us) server_negotiation.cc:407] Connection header received
0620 20:13:53.795891 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:53.795893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:53.795924 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:53.795986 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:53.796323 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43066: BlockingRecv error: recv got EOF from 127.0.0.1:43066 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 20:13:53.796748 24658 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/krb5cc)
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:53.797752 24658 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:53.798677 24658 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.129@KRBTEST.COM
I20260620 20:13:53.807013 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:53.799525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.24.20.129:46619)
0620 20:13:53.799644 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:53.799648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:53.799662 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:13:53.799752 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:53.799754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:53.799791 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:53.799866 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:53.800381 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.801016 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.801752 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.801979 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.802061 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:53.804196 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:53.804199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:53.804201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:53.804220 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:53.805754 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.806115 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.806118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.806119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.806183 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.806397 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.806399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.806400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.806537 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:53.806626 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:53.806821 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 20:13:53.806905 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 20:13:53.809433 30713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:13:53.808544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.24.20.129:46619)
0620 20:13:53.808679 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:53.808683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:53.808844 (+ 161us) server_negotiation.cc:407] Connection header received
0620 20:13:53.808919 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:53.808921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:53.808953 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:53.809022 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:53.809325 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43078: BlockingRecv error: recv got EOF from 127.0.0.1:43078 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 20:13:53.809566 24658 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986405628821-24658-0/krb5kdc/krb5cc' of type FILE
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:53.810544 24658 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:53.811447 24658 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.129@KRBTEST.COM
I20260620 20:13:53.820438 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:53.812319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43084 (local address 127.24.20.129:46619)
0620 20:13:53.812439 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:53.812444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:53.812539 (+ 95us) server_negotiation.cc:407] Connection header received
0620 20:13:53.812742 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:53.812745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:53.812783 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:53.812877 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:53.813383 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.813918 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.814749 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:53.815004 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:53.815116 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:53.817568 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:53.817571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:53.817573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:53.817594 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:53.818797 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.819233 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.819239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.819241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.819324 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:53.819619 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:53.819623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:53.819625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:53.819786 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:53.819923 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:53.820178 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 20:13:53.820292 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:13:53.821473 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 30309
I20260620 20:13:53.827155 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 30443
I20260620 20:13:53.832103 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 30577
I20260620 20:13:53.838199 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 30215
2026-06-20T20:13:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2136 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30726](info): set up 2 sockets
Jun 20 20:13:53 dist-test-slave-pflr krb5kdc[30726](info): commencing operation
krb5kdc: starting...
W20260620 20:13:54.869302 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:13:54 dist-test-slave-pflr krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:13:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.190.keytab.
Entry for principal oryx/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.190.keytab.
I20260620 20:13:54.985946 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:44703
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.190.keytab
--principal=oryx/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:44703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:55.094545 30739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:55.094791 30739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:55.094832 30739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:55.098842 30739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:55.098898 30739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:55.098918 30739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:55.098934 30739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:55.098949 30739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:55.104231 30739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:44703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.190.keytab
--principal=oryx/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:44703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:55.105325 30739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:55.106320 30739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:55.114373 30744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:55.114411 30745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:55.114411 30747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:55.114836 30739 server_base.cc:1034] running on GCE node
I20260620 20:13:55.115361 30739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:55.116523 30739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:55.117691 30739 hybrid_clock.cc:648] HybridClock initialized: now 1781986435117674 us; error 31 us; skew 500 ppm
Jun 20 20:13:55 dist-test-slave-pflr krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986435, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:55.125032 30739 init.cc:377] Logged in from keytab as oryx/127.24.20.190@KRBTEST.COM (short username oryx)
I20260620 20:13:55.126004 30739 webserver.cc:460] Webserver started at http://127.24.20.190:43967/ using document root <none> and password file <none>
I20260620 20:13:55.126550 30739 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:55.126609 30739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:55.126822 30739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:55.129014 30739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "c0bf847e90b54356bcdd594d09491430"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "e0ecba7bf7394e0bc986824e14656b02"
server_key_iv: "534e3b8f152c9cbffef45337318b35dc"
server_key_version: "encryptionkey@0"
I20260620 20:13:55.129565 30739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "c0bf847e90b54356bcdd594d09491430"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "e0ecba7bf7394e0bc986824e14656b02"
server_key_iv: "534e3b8f152c9cbffef45337318b35dc"
server_key_version: "encryptionkey@0"
I20260620 20:13:55.133172 30739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:13:55.135496 30754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:55.136760 30739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:55.136888 30739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "c0bf847e90b54356bcdd594d09491430"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "e0ecba7bf7394e0bc986824e14656b02"
server_key_iv: "534e3b8f152c9cbffef45337318b35dc"
server_key_version: "encryptionkey@0"
I20260620 20:13:55.136987 30739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:55.158296 30739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:55.161463 30739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:55.161659 30739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:55.168717 30739 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:44703
I20260620 20:13:55.169008 30806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:44703 every 8 connection(s)
I20260620 20:13:55.169829 30739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:13:55.171490 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 30739
I20260620 20:13:55.171594 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:13:55.171911 24658 external_mini_cluster.cc:1428] Setting key cac69051dd136421e3aca8643e4f4128
I20260620 20:13:55.172778 30807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:55.178306 30807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430: Bootstrap starting.
Jun 20 20:13:55 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
I20260620 20:13:55.180895 30807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:55.181774 30807 log.cc:824] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:55.183991 30807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430: No bootstrap required, opened a new log
I20260620 20:13:55.186218 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:55.173305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.24.20.190:44703)
0620 20:13:55.173725 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:55.173740 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:55.173774 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:13:55.174432 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:55.174449 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:55.174770 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:55.175096 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:55.175857 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.176911 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.177734 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.178193 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.178396 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:55.181093 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:55.181099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:55.181119 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:55.181155 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:55.183037 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:55.183452 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:55.183458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:55.183463 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:55.183604 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:55.183914 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:55.183917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:55.183918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:55.184454 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:55.184683 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:55.184976 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 20:13:55.185274 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":151,"threads_started":1}
I20260620 20:13:55.188449 30807 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } }
I20260620 20:13:55.188745 30807 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:55.188791 30807 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0bf847e90b54356bcdd594d09491430, State: Initialized, Role: FOLLOWER
I20260620 20:13:55.189189 30807 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } }
I20260620 20:13:55.189306 30807 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:13:55.189349 30807 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:13:55.189417 30807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:55.190411 30807 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } }
I20260620 20:13:55.190690 30807 leader_election.cc:302] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0bf847e90b54356bcdd594d09491430; no voters:
I20260620 20:13:55.190920 30807 leader_election.cc:288] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:13:55.191062 30812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:55.191291 30812 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [term 1 LEADER]: Becoming Leader. State: Replica: c0bf847e90b54356bcdd594d09491430, State: Running, Role: LEADER
I20260620 20:13:55.191577 30812 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } }
I20260620 20:13:55.192095 30807 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:55.193225 30813 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } }
I20260620 20:13:55.193274 30814 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0bf847e90b54356bcdd594d09491430. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } }
I20260620 20:13:55.193393 30813 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:55.193394 30814 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:55.193868 30821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:55.196817 30821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:55.201699 30821 catalog_manager.cc:1347] Generated new cluster ID: 4ba2f43188c24306b8eda927b1c49d9b
I20260620 20:13:55.201758 30821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:55.223986 30821 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:13:55.225293 30821 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:55.229786 30821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430: Generated new TSK 0
I20260620 20:13:55.230551 30821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab.
Entry for principal oryx/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab.
I20260620 20:13:55.260157 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44703
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab
--principal=oryx/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:55.368908 30833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:55.369148 30833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:55.369189 30833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:55.373044 30833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:55.373090 30833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:55.373142 30833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:13:55.378144 30833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab
--principal=oryx/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:55.379033 30833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:55.379920 30833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:55.388895 30838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:55.388871 30839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:55.388880 30841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:55.388921 30833 server_base.cc:1034] running on GCE node
I20260620 20:13:55.389523 30833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:55.390122 30833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:55.391281 30833 hybrid_clock.cc:648] HybridClock initialized: now 1781986435391255 us; error 38 us; skew 500 ppm
Jun 20 20:13:55 dist-test-slave-pflr krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986435, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:55.394178 30833 init.cc:377] Logged in from keytab as oryx/127.24.20.129@KRBTEST.COM (short username oryx)
I20260620 20:13:55.395008 30833 webserver.cc:460] Webserver started at http://127.24.20.129:39037/ using document root <none> and password file <none>
I20260620 20:13:55.395543 30833 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:55.395630 30833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:55.395845 30833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:55.397810 30833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "d1a7fa0094974f778502f3d8de7c5835"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "ff6d002f19e0abf3530d73e14e09331b"
server_key_iv: "0b6cc35bc7837c23999963b0dc5896a3"
server_key_version: "encryptionkey@0"
I20260620 20:13:55.398367 30833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "d1a7fa0094974f778502f3d8de7c5835"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "ff6d002f19e0abf3530d73e14e09331b"
server_key_iv: "0b6cc35bc7837c23999963b0dc5896a3"
server_key_version: "encryptionkey@0"
I20260620 20:13:55.401926 30833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:13:55.404258 30848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:55.405484 30833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:55.405642 30833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "d1a7fa0094974f778502f3d8de7c5835"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "ff6d002f19e0abf3530d73e14e09331b"
server_key_iv: "0b6cc35bc7837c23999963b0dc5896a3"
server_key_version: "encryptionkey@0"
I20260620 20:13:55.405752 30833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:55.423554 30833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:55.426213 30833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:55.426406 30833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:55.426949 30833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:55.427747 30833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:55.427819 30833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:55.427883 30833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:55.427932 30833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:55.437338 30833 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:41279
I20260620 20:13:55.437397 30961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:41279 every 8 connection(s)
I20260620 20:13:55.438181 30833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:55.446170 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 30833
I20260620 20:13:55.446318 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:13:55.446607 24658 external_mini_cluster.cc:1428] Setting key d5472a0533ca81d9792759cb64231931
Jun 20 20:13:55 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986435, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
I20260620 20:13:55.451809 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:55.439908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:49511 (local address 127.24.20.190:44703)
0620 20:13:55.440068 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:55.440071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:55.440897 (+ 826us) server_negotiation.cc:407] Connection header received
0620 20:13:55.442320 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:55.442323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:55.442380 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:55.442493 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:55.443579 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.444103 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.445296 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.445535 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.445646 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:55.448295 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:55.448298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:55.448300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:55.448320 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:55.449890 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:55.450369 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:55.450375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:55.450379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:55.450458 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:55.450778 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:55.450783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:55.450785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:55.450949 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:55.451056 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:55.451521 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 20:13:55.451638 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:13:55.452502 30964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:55.440279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.129:49511)
0620 20:13:55.440762 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:55.440802 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:55.442110 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:55.442663 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:55.442670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:55.442992 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:55.443431 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:55.443444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.444228 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.444232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:55.445166 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:55.445175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.445680 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.445684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:55.445849 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:55.446459 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:55.446475 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:55.448144 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:55.450013 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:55.450024 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:55.450034 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:55.450264 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:55.450553 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:55.450557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:55.450559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:55.450684 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:55.451070 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:55.451077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:55.451287 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 20:13:55.451462 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 20:13:55.451750 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
I20260620 20:13:55.453676 30962 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44703
I20260620 20:13:55.453919 30962 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:55.454466 30962 heartbeater.cc:511] Master 127.24.20.190:44703 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:13:55.456095 30771 ts_manager.cc:194] Registered new tserver with Master: d1a7fa0094974f778502f3d8de7c5835 (127.24.20.129:41279)
I20260620 20:13:55.457340 30771 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:49511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.20.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab.
Entry for principal oryx/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab.
I20260620 20:13:55.473703 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44703
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab
--principal=oryx/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:55.584657 30967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:55.584885 30967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:55.584926 30967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:55.589138 30967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:55.589187 30967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:55.589243 30967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:13:55.594520 30967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab
--principal=oryx/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:55.595453 30967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:55.596370 30967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:55.606022 30972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:55.606055 30975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:55.606091 30973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:55.606298 30967 server_base.cc:1034] running on GCE node
I20260620 20:13:55.606798 30967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:55.607368 30967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:55.608536 30967 hybrid_clock.cc:648] HybridClock initialized: now 1781986435608502 us; error 45 us; skew 500 ppm
Jun 20 20:13:55 dist-test-slave-pflr krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986435, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:55.611248 30967 init.cc:377] Logged in from keytab as oryx/127.24.20.130@KRBTEST.COM (short username oryx)
I20260620 20:13:55.612051 30967 webserver.cc:460] Webserver started at http://127.24.20.130:33667/ using document root <none> and password file <none>
I20260620 20:13:55.612630 30967 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:55.612725 30967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:55.612941 30967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:55.614874 30967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "bb509074a785475ea0930e20cc2a2107"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "88050ab90711c09a7a730737341640c0"
server_key_iv: "b76ac91b9b5d0d72c582162e9bf4432f"
server_key_version: "encryptionkey@0"
I20260620 20:13:55.615415 30967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "bb509074a785475ea0930e20cc2a2107"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "88050ab90711c09a7a730737341640c0"
server_key_iv: "b76ac91b9b5d0d72c582162e9bf4432f"
server_key_version: "encryptionkey@0"
I20260620 20:13:55.618887 30967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 20:13:55.621212 30982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:55.622387 30967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:13:55.622525 30967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "bb509074a785475ea0930e20cc2a2107"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "88050ab90711c09a7a730737341640c0"
server_key_iv: "b76ac91b9b5d0d72c582162e9bf4432f"
server_key_version: "encryptionkey@0"
I20260620 20:13:55.622623 30967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:55.646497 30967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:55.649288 30967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:55.649482 30967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:55.650039 30967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:55.650815 30967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:55.650885 30967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:55.650956 30967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:55.651006 30967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:55.660413 30967 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:44783
I20260620 20:13:55.660437 31095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:44783 every 8 connection(s)
I20260620 20:13:55.661283 30967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:55.669607 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 30967
I20260620 20:13:55.669746 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:13:55.670087 24658 external_mini_cluster.cc:1428] Setting key a22f20932d3beab050592d1d1e3c6aea
Jun 20 20:13:55 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986435, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
I20260620 20:13:55.675038 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:55.663062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:42923 (local address 127.24.20.190:44703)
0620 20:13:55.663189 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:55.663193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:55.664040 (+ 847us) server_negotiation.cc:407] Connection header received
0620 20:13:55.665462 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:55.665465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:55.665524 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:55.665598 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:55.666726 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.667268 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.668373 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.668646 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.668766 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:55.671409 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:55.671412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:55.671414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:55.671453 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:55.673095 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:55.673591 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:55.673595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:55.673597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:55.673679 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:55.674000 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:55.674005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:55.674007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:55.674190 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:55.674300 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:55.674756 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 20:13:55.674874 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:13:55.675840 31098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:55.663428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.130:42923)
0620 20:13:55.663900 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:55.663941 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:55.665242 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:55.665759 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:55.665766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:55.666096 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:55.666581 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:55.666594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.667389 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.667393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:55.668249 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:55.668257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.668794 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.668798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:55.668964 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:55.669534 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:55.669561 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:55.671222 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:55.673203 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:55.673207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:55.673217 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:55.673482 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:55.673779 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:55.673783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:55.673790 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:55.673906 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:55.674295 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:55.674301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:55.674517 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 20:13:55.674711 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 20:13:55.674998 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20260620 20:13:55.676952 31096 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44703
I20260620 20:13:55.677187 31096 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:55.677624 31096 heartbeater.cc:511] Master 127.24.20.190:44703 requested a full tablet report, sending...
I20260620 20:13:55.678618 30771 ts_manager.cc:194] Registered new tserver with Master: bb509074a785475ea0930e20cc2a2107 (127.24.20.130:44783)
WARNING: no policy specified for oryx/127.24.20.131@KRBTEST.COM; defaulting to no policy
I20260620 20:13:55.679445 30771 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:42923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.20.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab.
Entry for principal oryx/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab.
I20260620 20:13:55.695763 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44703
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab
--principal=oryx/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:55.801635 31101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:55.801877 31101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:55.801976 31101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:55.806044 31101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:55.806100 31101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:55.806157 31101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:13:55.811317 31101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab
--principal=oryx/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:55.812222 31101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:55.813195 31101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:55.822780 31107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:55.822856 31101 server_base.cc:1034] running on GCE node
W20260620 20:13:55.822801 31106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:55.822765 31109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:55.823355 31101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:55.824007 31101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:55.825166 31101 hybrid_clock.cc:648] HybridClock initialized: now 1781986435825154 us; error 27 us; skew 500 ppm
Jun 20 20:13:55 dist-test-slave-pflr krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986435, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:55.827867 31101 init.cc:377] Logged in from keytab as oryx/127.24.20.131@KRBTEST.COM (short username oryx)
I20260620 20:13:55.828712 31101 webserver.cc:460] Webserver started at http://127.24.20.131:40439/ using document root <none> and password file <none>
I20260620 20:13:55.829223 31101 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:55.829280 31101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:55.829443 31101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:55.831368 31101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "b6deecfe720547eca4dd1bd60cd96add"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "7eb15c318315bc3f5f5b8b047f5ae39e"
server_key_iv: "4d263d10f651c618070843efb5530dd3"
server_key_version: "encryptionkey@0"
I20260620 20:13:55.831856 31101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "b6deecfe720547eca4dd1bd60cd96add"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "7eb15c318315bc3f5f5b8b047f5ae39e"
server_key_iv: "4d263d10f651c618070843efb5530dd3"
server_key_version: "encryptionkey@0"
I20260620 20:13:55.835465 31101 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:13:55.837797 31116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:55.839017 31101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:55.839154 31101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "b6deecfe720547eca4dd1bd60cd96add"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "7eb15c318315bc3f5f5b8b047f5ae39e"
server_key_iv: "4d263d10f651c618070843efb5530dd3"
server_key_version: "encryptionkey@0"
I20260620 20:13:55.839264 31101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:55.866897 31101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:55.870234 31101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:55.870437 31101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:55.871008 31101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:55.871796 31101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:55.871869 31101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:55.871939 31101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:55.871982 31101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:55.880448 31101 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:44259
I20260620 20:13:55.880481 31229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:44259 every 8 connection(s)
I20260620 20:13:55.881325 31101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
Jun 20 20:13:55 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986435, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
I20260620 20:13:55.891606 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 31101
I20260620 20:13:55.891770 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:13:55.892053 24658 external_mini_cluster.cc:1428] Setting key 549b761ba93f96157571a12e5570c9b4
I20260620 20:13:55.894438 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:55.883100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:60117 (local address 127.24.20.190:44703)
0620 20:13:55.883250 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:55.883255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:55.884017 (+ 762us) server_negotiation.cc:407] Connection header received
0620 20:13:55.885232 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:55.885234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:55.885278 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:55.885354 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:55.886427 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.886933 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.888006 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.888214 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.888289 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:55.890558 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:55.890569 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:55.890571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:55.890593 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:55.892177 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:55.892778 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:55.892782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:55.892786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:55.892882 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:55.893288 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:55.893294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:55.893299 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:55.893530 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:55.893663 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:55.894184 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 20:13:55.894313 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:13:55.895223 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:55.883396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.131:60117)
0620 20:13:55.883853 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:55.883894 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:55.885052 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:55.885517 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:55.885523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:55.885854 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:55.886312 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:55.886325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.887037 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.887040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:55.887893 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:55.887902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:55.888335 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:55.888337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:55.888472 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:55.889027 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:55.889042 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:55.890402 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:55.892289 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:55.892294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:55.892307 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:55.892663 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:55.893000 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:55.893004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:55.893006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:55.893178 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:55.893645 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:55.893651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:55.893954 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 20:13:55.894140 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 20:13:55.894423 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":110,"threads_started":1}
I20260620 20:13:55.896131 31230 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44703
I20260620 20:13:55.896361 31230 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:55.896860 31230 heartbeater.cc:511] Master 127.24.20.190:44703 requested a full tablet report, sending...
I20260620 20:13:55.897764 30771 ts_manager.cc:194] Registered new tserver with Master: b6deecfe720547eca4dd1bd60cd96add (127.24.20.131:44259)
I20260620 20:13:55.898258 30771 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:60117
I20260620 20:13:55.906026 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.20.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.20.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.20.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.189.keytab.
Entry for principal oryx/127.24.20.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.189.keytab.
I20260620 20:13:55.931496 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.189:39307
--webserver_interface=127.24.20.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.189.keytab
--principal=oryx/127.24.20.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:44703,127.24.20.189:39307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:13:56.037137 31235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:56.037395 31235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:56.037470 31235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:56.041436 31235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:56.041527 31235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:56.041550 31235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:56.041566 31235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:56.041590 31235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:56.046680 31235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:44703,127.24.20.189:39307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.189.keytab
--principal=oryx/127.24.20.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.189:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.20.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
W20260620 20:13:56.046840 31235 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.20.190:44703,127.24.20.189:39307'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 20:13:56.047592 31235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:56.048480 31235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:13:56.056408 31235 server_base.cc:1034] running on GCE node
W20260620 20:13:56.056304 31243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:56.056299 31240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:56.056301 31241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:56.057035 31235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:56.058008 31235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:56.059165 31235 hybrid_clock.cc:648] HybridClock initialized: now 1781986436059153 us; error 23 us; skew 500 ppm
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:56.061755 31235 init.cc:377] Logged in from keytab as oryx/127.24.20.189@KRBTEST.COM (short username oryx)
I20260620 20:13:56.062618 31235 webserver.cc:460] Webserver started at http://127.24.20.189:35117/ using document root <none> and password file <none>
I20260620 20:13:56.063124 31235 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:56.063211 31235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:56.063426 31235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:56.065392 31235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/data/instance:
uuid: "bab681a79f3d420381d8a9edc451a543"
format_stamp: "Formatted at 2026-06-20 20:13:56 on dist-test-slave-pflr"
server_key: "9aafea78425c47fcd412c1af3bbf7348"
server_key_iv: "490e6910c03091aee4ee9e3075705d1e"
server_key_version: "encryptionkey@0"
I20260620 20:13:56.065935 31235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/wal/instance:
uuid: "bab681a79f3d420381d8a9edc451a543"
format_stamp: "Formatted at 2026-06-20 20:13:56 on dist-test-slave-pflr"
server_key: "9aafea78425c47fcd412c1af3bbf7348"
server_key_iv: "490e6910c03091aee4ee9e3075705d1e"
server_key_version: "encryptionkey@0"
I20260620 20:13:56.069317 31235 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:13:56.071485 31250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:56.072692 31235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 20:13:56.072814 31235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/wal
uuid: "bab681a79f3d420381d8a9edc451a543"
format_stamp: "Formatted at 2026-06-20 20:13:56 on dist-test-slave-pflr"
server_key: "9aafea78425c47fcd412c1af3bbf7348"
server_key_iv: "490e6910c03091aee4ee9e3075705d1e"
server_key_version: "encryptionkey@0"
I20260620 20:13:56.072916 31235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:56.089021 31235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:56.091619 31235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:56.091809 31235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:56.098531 31235 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.189:39307
I20260620 20:13:56.098804 31302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.189:39307 every 8 connection(s)
I20260620 20:13:56.099454 31235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/data/info.pb
I20260620 20:13:56.102008 31303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:56.105536 31303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:13:56.106770 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 31235
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.189@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.20.189@KRBTEST.COM
I20260620 20:13:56.120083 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.107452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.24.20.190:44703)
0620 20:13:56.107612 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.107617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.108527 (+ 910us) server_negotiation.cc:407] Connection header received
0620 20:13:56.109290 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.109293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.109339 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.109422 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.110722 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.111481 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.112991 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.113298 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.113411 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.116180 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.116184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.116187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.116217 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.117998 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.118569 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.118574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.118579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.118665 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.119013 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.119018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.119020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.119234 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.119349 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.119796 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.119908 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:13:56.120941 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.107919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.24.20.189:39307)
0620 20:13:56.108290 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.108303 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.108333 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:13:56.109051 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.109068 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.109294 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.109464 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.110410 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.111720 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.112862 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.113376 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.113515 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.116689 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.116695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.116704 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.116745 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.118501 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.118914 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.118920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.118921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.118999 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.119280 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.119285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.119287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.119579 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.119783 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.119994 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.120125 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":247,"spinlock_wait_cycles":22400,"thread_start_us":120,"threads_started":1}
I20260620 20:13:56.120944 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.107919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:38138)
0620 20:13:56.108387 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.108431 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.109079 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.109897 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.109904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.110365 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.110590 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.110600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.111616 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.111619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.112884 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.112893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.113498 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.113503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.113620 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.114220 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.114241 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.115988 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.118107 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.118113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.118120 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.118470 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.118773 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.118776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.118778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.118906 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.119347 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.119354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.119589 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.119753 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.120045 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":1,"spinlock_wait_cycles":15744,"thread_start_us":162,"threads_started":1}
I20260620 20:13:56.122222 31303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.189@KRBTEST.COM for oryx/127.24.20.189@KRBTEST.COM
I20260620 20:13:56.130288 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.122677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39307 (local address 127.0.0.1:54932)
0620 20:13:56.122819 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.122834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.122903 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.123124 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.123127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.123274 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.123345 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.123349 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.124102 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.124107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.124960 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.124968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.125837 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.125840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.125952 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.126329 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.126339 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.127521 (+ 1182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.129161 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.129164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.129166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.129369 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.129587 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.129589 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.129590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.129657 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.129935 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.129937 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.130001 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.130075 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.130167 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 20:13:56.130368 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.122779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.24.20.189:39307)
0620 20:13:56.122887 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.122891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.122902 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:13:56.122996 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.122998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.123030 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.123114 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.123457 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.123997 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.125066 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.125728 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.125842 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.127633 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.127636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.127638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.127656 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.129061 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.129450 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.129452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.129453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.129514 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.129726 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.129728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.129729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.129860 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.129929 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.130104 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.130227 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":48}
I20260620 20:13:56.132915 31303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: Bootstrap starting.
I20260620 20:13:56.134912 31303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:56.135537 31303 log.cc:824] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:56.137267 31303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: No bootstrap required, opened a new log
I20260620 20:13:56.140071 31303 raft_consensus.cc:348] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } }
I20260620 20:13:56.140244 31303 raft_consensus.cc:374] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:56.140277 31303 raft_consensus.cc:729] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab681a79f3d420381d8a9edc451a543, State: Initialized, Role: FOLLOWER
I20260620 20:13:56.140671 31303 consensus_queue.cc:260] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } }
I20260620 20:13:56.141161 31309 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } } }
I20260620 20:13:56.141285 31309 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:13:56.141791 31303 sys_catalog.cc:564] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:13:56.146471 31320 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:13:56.146575 31320 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:13:56.148176 31235 master_runner.cc:428] Detected that this master bab681a79f3d420381d8a9edc451a543 is joining an existing cluster
I20260620 20:13:56.148254 31235 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.20.189:39307
I20260620 20:13:56.158999 31326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.150183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:38142)
0620 20:13:56.150490 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.150505 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.150652 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.151037 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.151040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.151246 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.151523 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.151531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.152301 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.152305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.153250 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.153259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.153865 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.153868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.154034 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.154776 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.154793 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.155258 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.157418 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.157423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.157425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.157684 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.158016 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.158019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.158021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.158107 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.158480 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.158483 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.158582 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.158688 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.158817 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":95,"spinlock_wait_cycles":10624,"thread_start_us":83,"threads_started":1}
I20260620 20:13:56.159062 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.150232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.24.20.190:44703)
0620 20:13:56.150347 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.150352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.150570 (+ 218us) server_negotiation.cc:407] Connection header received
0620 20:13:56.150771 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.150776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.150823 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.150877 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.151675 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.152180 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.153384 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.153676 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.153733 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.155417 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.155420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.155425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.155451 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.157308 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.157804 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.157812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.157813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.157918 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.158219 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.158226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.158227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.158393 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.158469 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.158762 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.158876 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:13:56.159286 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.150559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54938 (local address 127.24.20.189:39307)
0620 20:13:56.150673 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.150678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.150691 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:13:56.150792 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.150794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.150836 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.150917 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.151642 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.152403 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.153507 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.153840 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.153948 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.155401 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.155405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.155408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.155431 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.157344 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.157876 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.157881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.157882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.157954 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.158506 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.158508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.158509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.158653 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.158767 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.159030 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.159140 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":48}
I20260620 20:13:56.159596 31327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.150268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39307 (local address 127.0.0.1:54938)
0620 20:13:56.150623 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.150634 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.150705 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.150950 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.150953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.151246 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.151523 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.151531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.152520 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.152523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.153401 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.153409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.153953 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.153956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.154058 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.154676 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.154686 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.155258 (+ 572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.157455 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.157460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.157463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.157751 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.158049 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.158051 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.158053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.158126 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.158799 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.158801 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.158882 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.159321 (+ 439us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.159411 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":74,"thread_start_us":84,"threads_started":1}
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986435, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.190@KRBTEST.COM for oryx/127.24.20.189@KRBTEST.COM
I20260620 20:13:56.172159 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.163464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.24.20.189:39307)
0620 20:13:56.163565 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.163569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.163855 (+ 286us) server_negotiation.cc:407] Connection header received
0620 20:13:56.164048 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.164051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.164095 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.164188 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.164830 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.165316 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.166161 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.166408 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.166519 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.168847 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.168851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.168853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.168876 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.170419 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.170927 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.170930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.170932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.171008 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.171288 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.171291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.171292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.171478 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.171573 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.171905 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.172010 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 20:13:56.172178 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.163424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39307 (local address 127.0.0.1:54940)
0620 20:13:56.163746 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.163783 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.163928 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.164220 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.164225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.164563 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.164722 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.164728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.165424 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.165429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.166035 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.166044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.166520 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.166523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.166650 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.167125 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.167136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.168720 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.170533 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.170540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.170545 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.170828 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.171107 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.171110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.171112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.171199 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.171580 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.171586 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.171697 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.171841 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.172009 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
I20260620 20:13:56.173743 30771 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.20.189:39307: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bab681a79f3d420381d8a9edc451a543"
member_type: NON_VOTER
last_known_addr {
host: "127.24.20.189"
port: 39307
}
attrs {
promote: true
}
}
dest_uuid: "c0bf847e90b54356bcdd594d09491430"
cas_config_opid_index: -1
I20260620 20:13:56.174366 30771 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true } }
I20260620 20:13:56.176801 31329 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bab681a79f3d420381d8a9edc451a543 (127.24.20.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true } } }
I20260620 20:13:56.177196 31277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:56.178416 31329 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.20.189:39307
I20260620 20:13:56.178506 31277 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 FOLLOWER]: Refusing update from remote peer c0bf847e90b54356bcdd594d09491430: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 20:13:56.178704 31329 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true } } }
I20260620 20:13:56.178889 31329 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:56.179168 31235 master.cc:537] Master@127.24.20.189:39307 shutting down...
I20260620 20:13:56.179147 31329 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:13:56.180713 30757 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 20:13:56.181725 30757 consensus_peers.cc:487] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 -> Peer bab681a79f3d420381d8a9edc451a543 (127.24.20.189:39307): Couldn't send request to peer bab681a79f3d420381d8a9edc451a543. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 20:13:56.183636 31235 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:56.183934 31235 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:56.184079 31235 tablet_replica.cc:330] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: stopping tablet replica
WARNING: no policy specified for oryx/127.24.20.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.20.188@KRBTEST.COM" created.
I20260620 20:13:56.198880 31235 master.cc:552] Master@127.24.20.189:39307 shutdown complete.
I20260620 20:13:56.198932 31235 master_runner.cc:305] Clearing existing system tablet
I20260620 20:13:56.199180 31235 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:56.199234 31235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:56.200536 31235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:56.201524 31336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:56.201726 31235 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:56.201851 31235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:56.201884 31235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/wal
uuid: "bab681a79f3d420381d8a9edc451a543"
format_stamp: "Formatted at 2026-06-20 20:13:56 on dist-test-slave-pflr"
server_key: "9aafea78425c47fcd412c1af3bbf7348"
server_key_iv: "490e6910c03091aee4ee9e3075705d1e"
server_key_version: "encryptionkey@0"
I20260620 20:13:56.203992 31235 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:56.206832 31235 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 20:13:56.206952 31235 log.cc:1196] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 20:13:56.207630 31235 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: Deleting consensus metadata
I20260620 20:13:56.207792 31235 master_runner.cc:315] Copying system tablet from 127.24.20.190:44703
I20260620 20:13:56.208971 31235 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: tablet copy: Beginning tablet copy session from remote peer at address 127.24.20.190:44703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.20.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.188.keytab.
Entry for principal oryx/127.24.20.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.188.keytab.
I20260620 20:13:56.212328 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.188:45015
--webserver_interface=127.24.20.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.188.keytab
--principal=oryx/127.24.20.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:44703,127.24.20.189:39307,127.24.20.188:45015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:13:56.217880 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.209659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.24.20.190:44703)
0620 20:13:56.209790 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.209795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.209953 (+ 158us) server_negotiation.cc:407] Connection header received
0620 20:13:56.210062 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.210065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.210122 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.210196 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.210723 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.211345 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.212502 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.212909 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.213004 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.214570 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.214575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.214578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.214603 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.216166 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.216689 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.216697 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.216699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.216770 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.217061 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.217065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.217067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.217237 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.217351 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.217615 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.217724 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 20:13:56.217890 31342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.209586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:38148)
0620 20:13:56.209850 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.209865 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.209985 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.210234 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.210238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.210473 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.210601 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.210608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.211455 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.211466 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.212339 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.212352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.213184 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.213190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.213320 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.213939 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.213967 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.214419 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.216286 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.216292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.216295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.216538 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.216877 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.216880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.216882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.216972 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.217339 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.217342 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.217444 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.217551 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.217692 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":99,"threads_started":1}
I20260620 20:13:56.218636 30791 tablet_copy_service.cc:140] P c0bf847e90b54356bcdd594d09491430: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bab681a79f3d420381d8a9edc451a543 ({username='oryx', principal='oryx/127.24.20.189@KRBTEST.COM'} at 127.0.0.1:38148)
I20260620 20:13:56.218807 30791 tablet_copy_service.cc:161] P c0bf847e90b54356bcdd594d09491430: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bab681a79f3d420381d8a9edc451a543 at {username='oryx', principal='oryx/127.24.20.189@KRBTEST.COM'} at 127.0.0.1:38148: session id = bab681a79f3d420381d8a9edc451a543-00000000000000000000000000000000
I20260620 20:13:56.220263 30791 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 20:13:56.221732 31235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 20:13:56.223519 31235 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: tablet copy: Starting download of 0 data blocks...
I20260620 20:13:56.223718 31235 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: tablet copy: Starting download of 1 WAL segments...
I20260620 20:13:56.225282 31235 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 20:13:56.226816 30791 tablet_copy_service.cc:342] P c0bf847e90b54356bcdd594d09491430: Request end of tablet copy session bab681a79f3d420381d8a9edc451a543-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.20.189@KRBTEST.COM'} at 127.0.0.1:38148
I20260620 20:13:56.226938 30791 tablet_copy_service.cc:433] P c0bf847e90b54356bcdd594d09491430: ending tablet copy session bab681a79f3d420381d8a9edc451a543-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bab681a79f3d420381d8a9edc451a543
I20260620 20:13:56.233685 31235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:56.235311 31344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:56.235484 31345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:56.235774 31347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:56.236033 31235 server_base.cc:1034] running on GCE node
I20260620 20:13:56.236183 31235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:56.236425 31235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:56.237581 31235 hybrid_clock.cc:648] HybridClock initialized: now 1781986436237567 us; error 30 us; skew 500 ppm
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:56.239059 31235 init.cc:377] Logged in from keytab as oryx/127.24.20.189@KRBTEST.COM (short username oryx)
I20260620 20:13:56.239511 31235 webserver.cc:460] Webserver started at http://127.24.20.189:44113/ using document root <none> and password file <none>
I20260620 20:13:56.239688 31235 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:56.239740 31235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:56.241007 31235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:56.241921 31354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:56.242149 31235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:56.242221 31235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/wal
uuid: "bab681a79f3d420381d8a9edc451a543"
format_stamp: "Formatted at 2026-06-20 20:13:56 on dist-test-slave-pflr"
server_key: "9aafea78425c47fcd412c1af3bbf7348"
server_key_iv: "490e6910c03091aee4ee9e3075705d1e"
server_key_version: "encryptionkey@0"
I20260620 20:13:56.242350 31235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:56.252899 31235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:56.254846 31235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:56.259501 31235 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.189:39307
I20260620 20:13:56.260063 31235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-1/data/info.pb
I20260620 20:13:56.260468 31406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.189:39307 every 8 connection(s)
I20260620 20:13:56.261554 31407 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:13:56.262322 31407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: Bootstrap starting.
I20260620 20:13:56.269892 31407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:13:56.270167 31407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: Bootstrap complete.
I20260620 20:13:56.270610 31407 raft_consensus.cc:348] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true } }
I20260620 20:13:56.270738 31407 raft_consensus.cc:729] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bab681a79f3d420381d8a9edc451a543, State: Initialized, Role: LEARNER
I20260620 20:13:56.270850 31407 consensus_queue.cc:260] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true } }
I20260620 20:13:56.271160 31410 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true } } }
I20260620 20:13:56.271312 31410 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:13:56.271320 31407 sys_catalog.cc:564] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:56.273983 31421 catalog_manager.cc:1259] Loaded cluster ID: 4ba2f43188c24306b8eda927b1c49d9b
I20260620 20:13:56.274085 31421 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: loading cluster ID for follower catalog manager: success
I20260620 20:13:56.275699 31421 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: acquiring CA information for follower catalog manager: success
I20260620 20:13:56.276767 31421 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 20:13:56.352990 31343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:56.353250 31343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:56.353322 31343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:56.357147 31343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:13:56.357219 31343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:56.357262 31343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:56.357316 31343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:13:56.357350 31343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:13:56.362376 31343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:44703,127.24.20.189:39307,127.24.20.188:45015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.188.keytab
--principal=oryx/127.24.20.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.188:45015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.20.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:56.363325 31343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:56.364259 31343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:56.372242 31426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:56.372241 31427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:56.372378 31429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:56.372452 31343 server_base.cc:1034] running on GCE node
I20260620 20:13:56.373133 31343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:56.374116 31343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:56.375267 31343 hybrid_clock.cc:648] HybridClock initialized: now 1781986436375241 us; error 39 us; skew 500 ppm
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:56.377887 31343 init.cc:377] Logged in from keytab as oryx/127.24.20.188@KRBTEST.COM (short username oryx)
I20260620 20:13:56.378710 31343 webserver.cc:460] Webserver started at http://127.24.20.188:37887/ using document root <none> and password file <none>
I20260620 20:13:56.379218 31343 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:56.379276 31343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:56.379469 31343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:13:56.381394 31343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/data/instance:
uuid: "08260900a88b4eb59a21b4a0071e85f3"
format_stamp: "Formatted at 2026-06-20 20:13:56 on dist-test-slave-pflr"
server_key: "df2abd74d95524ee0acd1222bb7d8da0"
server_key_iv: "ccf888c02e9ab4ad7d9b62bb6941a3f7"
server_key_version: "encryptionkey@0"
I20260620 20:13:56.381923 31343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/wal/instance:
uuid: "08260900a88b4eb59a21b4a0071e85f3"
format_stamp: "Formatted at 2026-06-20 20:13:56 on dist-test-slave-pflr"
server_key: "df2abd74d95524ee0acd1222bb7d8da0"
server_key_iv: "ccf888c02e9ab4ad7d9b62bb6941a3f7"
server_key_version: "encryptionkey@0"
I20260620 20:13:56.385408 31343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:13:56.387529 31436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:56.388764 31343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:13:56.388854 31343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/wal
uuid: "08260900a88b4eb59a21b4a0071e85f3"
format_stamp: "Formatted at 2026-06-20 20:13:56 on dist-test-slave-pflr"
server_key: "df2abd74d95524ee0acd1222bb7d8da0"
server_key_iv: "ccf888c02e9ab4ad7d9b62bb6941a3f7"
server_key_version: "encryptionkey@0"
I20260620 20:13:56.388978 31343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:56.407862 31343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:56.411137 31343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:56.411324 31343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:56.418212 31343 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.188:45015
I20260620 20:13:56.418494 31488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.188:45015 every 8 connection(s)
I20260620 20:13:56.419152 31343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/data/info.pb
I20260620 20:13:56.421798 31489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:56.425287 31489 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:13:56.429301 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 31343
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.188@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.20.188@KRBTEST.COM
I20260620 20:13:56.439953 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.427126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.24.20.190:44703)
0620 20:13:56.427278 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.427283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.428058 (+ 775us) server_negotiation.cc:407] Connection header received
0620 20:13:56.428899 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.428902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.428948 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.429028 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.430287 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.431029 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.432658 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.432962 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.433081 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.435620 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.435629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.435632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.435655 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.437120 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.438118 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.438122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.438125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.438233 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.438690 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.438693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.438695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.438908 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.439023 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.439576 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.439690 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:13:56.440809 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.427467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:37188)
0620 20:13:56.427900 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.427938 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.428724 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.429163 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.429169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.429523 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.430115 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.430134 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.431177 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.431180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.432504 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.432514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.433096 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.433099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.433241 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.433816 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.433834 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.435483 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.437643 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.437650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.437658 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.438012 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.438336 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.438340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.438342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.438558 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.439044 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.439051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.439337 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.439532 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.439840 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":43,"thread_start_us":107,"threads_started":1}
I20260620 20:13:56.441208 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.430083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.24.20.188:45015)
0620 20:13:56.430356 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.430366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.430386 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:13:56.430436 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.430441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.430611 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.430757 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.431394 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.432683 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.433673 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.434154 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.434258 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.437294 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.437300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.437312 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.437335 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.439221 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.439706 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.439712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.439714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.439803 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.440144 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.440147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.440148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.440401 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.440631 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.440843 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.440984 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
I20260620 20:13:56.442389 31489 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.188@KRBTEST.COM for oryx/127.24.20.189@KRBTEST.COM
I20260620 20:13:56.450768 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.442891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39307 (local address 127.0.0.1:34990)
0620 20:13:56.442982 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.442996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.443065 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.443663 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.443666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.443817 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.443897 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.443902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.444890 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.444895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.445534 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.445549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.446181 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.446185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.446328 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.446763 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.446774 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.447857 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.449401 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.449406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.449408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.449592 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.449869 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.449873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.449875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.449956 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.450310 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.450313 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.450399 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.450505 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.450622 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 20:13:56.450834 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.443168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.24.20.189:39307)
0620 20:13:56.443414 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.443417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.443428 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:13:56.443463 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.443465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.443500 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.443646 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.444006 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.444768 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.445677 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.446068 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.446186 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.447986 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.447990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.447992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.448016 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.449297 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.449693 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.449698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.449700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.449778 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.450043 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.450048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.450050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.450218 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.450322 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.450564 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.450683 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
I20260620 20:13:56.451573 31489 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.188@KRBTEST.COM for oryx/127.24.20.188@KRBTEST.COM
I20260620 20:13:56.458649 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.452001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:45015 (local address 127.0.0.1:42942)
0620 20:13:56.452167 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.452182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.452258 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.452451 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.452453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.452658 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.452729 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.452734 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.453424 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.453427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.453906 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.453912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.454325 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.454328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.454424 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.454799 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.454809 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.455917 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.457327 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.457333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.457336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.457510 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.457772 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.457776 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.457778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.457853 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.458181 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.458184 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.458257 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.458368 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.458485 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":2}
I20260620 20:13:56.458673 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.452082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.24.20.188:45015)
0620 20:13:56.452170 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.452176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.452223 (+ 47us) server_negotiation.cc:407] Connection header received
0620 20:13:56.452324 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.452328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.452364 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.452441 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.452836 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.453305 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.454009 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.454229 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.454329 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.456046 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.456054 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.456056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.456074 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.457226 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.457600 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.457606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.457608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.457685 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.457937 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.457942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.457944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.458087 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.458190 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.458410 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.458535 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":40}
I20260620 20:13:56.459808 30962 heartbeater.cc:503] Master 127.24.20.190:44703 was elected leader, sending a full tablet report...
I20260620 20:13:56.462071 31489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: Bootstrap starting.
I20260620 20:13:56.464231 31489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:56.464995 31489 log.cc:824] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:56.466758 31489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: No bootstrap required, opened a new log
I20260620 20:13:56.469682 31489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } }
I20260620 20:13:56.469899 31489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:56.469983 31489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08260900a88b4eb59a21b4a0071e85f3, State: Initialized, Role: FOLLOWER
I20260620 20:13:56.470424 31489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } }
I20260620 20:13:56.470877 31495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } } }
I20260620 20:13:56.470984 31495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:13:56.471601 31489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:13:56.476181 31506 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:13:56.476264 31506 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:13:56.479214 31343 master_runner.cc:428] Detected that this master 08260900a88b4eb59a21b4a0071e85f3 is joining an existing cluster
I20260620 20:13:56.479272 31343 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.20.188:45015
I20260620 20:13:56.490377 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.481485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.24.20.190:44703)
0620 20:13:56.481604 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.481609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.482232 (+ 623us) server_negotiation.cc:407] Connection header received
0620 20:13:56.482395 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.482400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.482468 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.482543 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.483326 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.484093 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.485392 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.485727 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.485799 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.487334 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.487338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.487340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.487370 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.488820 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.489241 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.489244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.489246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.489323 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.489597 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.489602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.489604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.489769 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.489874 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.490132 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.490238 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 20:13:56.490397 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.481442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:37202)
0620 20:13:56.482156 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.482170 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.482286 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.482678 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.482680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.483001 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.483198 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.483203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.484211 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.484216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.485263 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.485269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.485920 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.485923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.486354 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.486826 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.486837 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.487193 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.488937 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.488941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.488944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.489137 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.489418 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.489421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.489423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.489510 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.489863 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.489866 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.489971 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.490077 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.490208 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":467,"spinlock_wait_cycles":768,"thread_start_us":140,"threads_started":1}
I20260620 20:13:56.494220 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 30833
I20260620 20:13:56.502353 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:41279
--local_ip_for_outbound_sockets=127.24.20.129
--tserver_master_addrs=127.24.20.190:44703
--webserver_port=39037
--webserver_interface=127.24.20.129
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab
--principal=oryx/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.20.190:44703,127.24.20.189:39307
--tserver_master_addrs=127.24.20.190:44703,127.24.20.189:39307,127.24.20.188:45015 with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986435, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.190@KRBTEST.COM for oryx/127.24.20.188@KRBTEST.COM
I20260620 20:13:56.513195 31512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.481449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39307 (local address 127.0.0.1:34998)
0620 20:13:56.481938 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.481952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.482093 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.482455 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.482462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.482712 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.482839 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.482846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.483945 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.483948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.484900 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.484910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.485670 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.485674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.485820 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.486315 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.486330 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.509095 (+ 22765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.511241 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.511245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.511247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.511521 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.511832 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.511835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.511837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.511917 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.512652 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.512654 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.512756 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.512855 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.512997 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":26,"thread_start_us":90,"threads_started":1}
I20260620 20:13:56.514564 31514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.481946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:45015 (local address 127.0.0.1:42950)
0620 20:13:56.482347 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.482361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.482471 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.482751 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.482753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.483013 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.483194 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.483202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.484171 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.484176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.485255 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.485265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.485920 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.485923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.486074 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.509105 (+ 23031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.509120 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.509516 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.512348 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.512351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.512353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.512575 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.513678 (+ 1103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.513681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.513683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.513752 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.514117 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.514120 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.514189 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.514267 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.514389 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":109,"spinlock_wait_cycles":3584,"thread_start_us":79,"threads_started":1}
I20260620 20:13:56.515105 31492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.481989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.24.20.188:45015)
0620 20:13:56.482078 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.482082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.482429 (+ 347us) server_negotiation.cc:407] Connection header received
0620 20:13:56.482563 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.482566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.482597 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.482640 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.483313 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.484050 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.485376 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.485677 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.485744 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.509676 (+ 23932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.509683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.509689 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.509721 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.511972 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.513500 (+ 1528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.513503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.513505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.513577 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.513830 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.513833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.513835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.514008 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.514088 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.514840 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.514966 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":35}
I20260620 20:13:56.515303 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.481670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.24.20.189:39307)
0620 20:13:56.481786 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.481791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.482026 (+ 235us) server_negotiation.cc:407] Connection header received
0620 20:13:56.482186 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.482190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.482241 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.482321 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.482973 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.483801 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.485031 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.485415 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.485493 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.509285 (+ 23792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.509292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.509297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.509334 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.511117 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.511642 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.511645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.511647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.511731 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.512008 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.512011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.512013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.512222 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.512308 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.515007 (+ 2699us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.515153 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:13:56.517915 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.494314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:45015 (local address 127.0.0.1:42966)
0620 20:13:56.494895 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.494907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.494969 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.497111 (+ 2142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.497114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.497299 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.497396 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.497402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.498512 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.498515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.499363 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.499374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.500008 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.500012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.500109 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.501433 (+ 1324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.501453 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.502893 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.516266 (+ 13373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.516271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.516275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.516526 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.516879 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.516883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.516885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.516987 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.517392 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.517394 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.517499 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.517599 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.517752 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517}
I20260620 20:13:56.517937 31515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.494515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.24.20.188:45015)
0620 20:13:56.496805 (+ 2290us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.496838 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.496853 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:13:56.496915 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.496918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.496976 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.497500 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.497596 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.498388 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.499503 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.499868 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.500351 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.503053 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.503056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.503059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.503082 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.516149 (+ 13067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.516692 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.516695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.516697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.516779 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.517099 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.517102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.517104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.517287 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.517370 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.517673 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.517781 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2203,"thread_start_us":134,"threads_started":1}
I20260620 20:13:56.518791 30771 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.20.188:45015: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3"
member_type: NON_VOTER
last_known_addr {
host: "127.24.20.188"
port: 45015
}
attrs {
promote: true
}
}
dest_uuid: "c0bf847e90b54356bcdd594d09491430"
cas_config_opid_index: 5
I20260620 20:13:56.519402 30771 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } }
I20260620 20:13:56.520408 31331 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 08260900a88b4eb59a21b4a0071e85f3 (127.24.20.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } } }
I20260620 20:13:56.521924 31331 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.20.188:45015
I20260620 20:13:56.522099 31331 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } } }
I20260620 20:13:56.522202 31331 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:56.522598 31343 master.cc:537] Master@127.24.20.188:45015 shutting down...
I20260620 20:13:56.523013 31463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:56.524470 31463 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 1 FOLLOWER]: Refusing update from remote peer c0bf847e90b54356bcdd594d09491430: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 20:13:56.525044 31329 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:13:56.526752 30755 consensus_peers.cc:487] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 -> Peer 08260900a88b4eb59a21b4a0071e85f3 (127.24.20.188:45015): Couldn't send request to peer 08260900a88b4eb59a21b4a0071e85f3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 20:13:56.528393 31343 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:56.528708 31343 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:56.528807 31343 tablet_replica.cc:330] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: stopping tablet replica
I20260620 20:13:56.532364 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.520541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39307 (local address 127.0.0.1:35002)
0620 20:13:56.527360 (+ 6819us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.527375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.527456 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.528284 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.528287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.528306 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:56.528393 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.528398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.529460 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.529463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.530856 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.530867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.531965 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.531969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.532061 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.532106 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.532178 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.532243 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6742}
I20260620 20:13:56.532922 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.520764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.24.20.189:39307)
0620 20:13:56.527605 (+ 6841us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.527609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.527621 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:13:56.527664 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.527667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.527723 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.527812 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:56.528520 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.529342 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.530994 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.531660 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.531760 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.531907 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.532709 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.532784 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6767}
I20260620 20:13:56.533545 31372 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 LEARNER]: Refusing update from remote peer c0bf847e90b54356bcdd594d09491430: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 20:13:56.533959 31329 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 20:13:56.535729 31372 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 08260900a88b4eb59a21b4a0071e85f3 (127.24.20.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } } }
I20260620 20:13:56.537223 31343 master.cc:552] Master@127.24.20.188:45015 shutdown complete.
I20260620 20:13:56.537286 31343 master_runner.cc:305] Clearing existing system tablet
I20260620 20:13:56.537530 31343 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:56.537473 31410 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0bf847e90b54356bcdd594d09491430. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } } }
I20260620 20:13:56.537595 31410 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:13:56.537619 31343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:56.537681 31410 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: NON_VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: true } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } } }
I20260620 20:13:56.537773 31410 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:13:56.538409 31329 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430: attempting to promote NON_VOTER bab681a79f3d420381d8a9edc451a543 to VOTER
I20260620 20:13:56.538832 31329 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } }
I20260620 20:13:56.538985 31343 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260620 20:13:56.539804 31521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:56.540015 31343 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:56.540117 31343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:56.540148 31343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/wal
uuid: "08260900a88b4eb59a21b4a0071e85f3"
format_stamp: "Formatted at 2026-06-20 20:13:56 on dist-test-slave-pflr"
server_key: "df2abd74d95524ee0acd1222bb7d8da0"
server_key_iv: "ccf888c02e9ab4ad7d9b62bb6941a3f7"
server_key_version: "encryptionkey@0"
I20260620 20:13:56.540273 31381 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 LEARNER]: Refusing update from remote peer c0bf847e90b54356bcdd594d09491430: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 20:13:56.540665 31331 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 20:13:56.541994 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.541703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:45015 (local address 127.0.0.1:42974)
0620 20:13:56.541844 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.541907 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:45015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 20:13:56.542271 31343 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:56.542755 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.542513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:45015 (local address 127.0.0.1:42990)
0620 20:13:56.542658 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.542701 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:45015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 20:13:56.542894 30755 consensus_peers.cc:487] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 -> Peer 08260900a88b4eb59a21b4a0071e85f3 (127.24.20.188:45015): Couldn't send request to peer 08260900a88b4eb59a21b4a0071e85f3. Status: Network error: Client connection negotiation failed: client connection to 127.24.20.188:45015: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 20:13:56.543787 31331 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bab681a79f3d420381d8a9edc451a543 (127.24.20.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } } }
I20260620 20:13:56.544438 31343 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 20:13:56.544507 31343 log.cc:1196] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 20:13:56.544795 31343 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: Deleting consensus metadata
I20260620 20:13:56.544941 31343 master_runner.cc:315] Copying system tablet from 127.24.20.190:44703
I20260620 20:13:56.545065 31331 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } } }
I20260620 20:13:56.545158 31331 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:56.545267 31331 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } } }
I20260620 20:13:56.545325 31331 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:56.545954 31343 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: tablet copy: Beginning tablet copy session from remote peer at address 127.24.20.190:44703
I20260620 20:13:56.545898 31381 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bab681a79f3d420381d8a9edc451a543 (127.24.20.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } } }
I20260620 20:13:56.547408 31518 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } } }
I20260620 20:13:56.547550 31518 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:13:56.555546 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.547063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:37210)
0620 20:13:56.547292 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.547305 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.547429 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.547846 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.547849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.548058 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.548164 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.548171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.549310 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.549314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.550217 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.550228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.550812 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.550816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.550912 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.551461 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.551476 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.551910 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.553973 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.553976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.553979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.554223 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.554526 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.554529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.554530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.554619 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.555029 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.555032 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.555125 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.555231 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.555357 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":54,"threads_started":1}
I20260620 20:13:56.555548 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.546832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.24.20.190:44703)
0620 20:13:56.546980 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.546984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.547365 (+ 381us) server_negotiation.cc:407] Connection header received
0620 20:13:56.547536 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.547539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.547597 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.547657 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.548294 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.549179 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.550351 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.550688 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.550800 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.552116 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.552120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.552123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.552147 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.553862 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.554329 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.554335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.554338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.554427 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.554719 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.554722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.554724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.554930 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.555029 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.555295 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.555409 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 20:13:56.556392 30791 tablet_copy_service.cc:140] P c0bf847e90b54356bcdd594d09491430: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 08260900a88b4eb59a21b4a0071e85f3 ({username='oryx', principal='oryx/127.24.20.188@KRBTEST.COM'} at 127.0.0.1:37210)
I20260620 20:13:56.556484 30791 tablet_copy_service.cc:161] P c0bf847e90b54356bcdd594d09491430: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 08260900a88b4eb59a21b4a0071e85f3 at {username='oryx', principal='oryx/127.24.20.188@KRBTEST.COM'} at 127.0.0.1:37210: session id = 08260900a88b4eb59a21b4a0071e85f3-00000000000000000000000000000000
I20260620 20:13:56.557507 30791 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 20:13:56.558698 31343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 20:13:56.560819 31343 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: tablet copy: Starting download of 0 data blocks...
I20260620 20:13:56.560992 31343 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: tablet copy: Starting download of 1 WAL segments...
I20260620 20:13:56.562337 31343 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 20:13:56.564406 30791 tablet_copy_service.cc:342] P c0bf847e90b54356bcdd594d09491430: Request end of tablet copy session 08260900a88b4eb59a21b4a0071e85f3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.20.188@KRBTEST.COM'} at 127.0.0.1:37210
I20260620 20:13:56.564482 30791 tablet_copy_service.cc:433] P c0bf847e90b54356bcdd594d09491430: ending tablet copy session 08260900a88b4eb59a21b4a0071e85f3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 08260900a88b4eb59a21b4a0071e85f3
I20260620 20:13:56.571493 31343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:56.573174 31529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:56.573197 31528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:56.573922 31531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:56.574046 31343 server_base.cc:1034] running on GCE node
I20260620 20:13:56.574267 31343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:56.574533 31343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:56.575681 31343 hybrid_clock.cc:648] HybridClock initialized: now 1781986436575668 us; error 29 us; skew 500 ppm
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:56.577443 31343 init.cc:377] Logged in from keytab as oryx/127.24.20.188@KRBTEST.COM (short username oryx)
I20260620 20:13:56.577874 31343 webserver.cc:460] Webserver started at http://127.24.20.188:44121/ using document root <none> and password file <none>
I20260620 20:13:56.578064 31343 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:56.578105 31343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:56.579345 31343 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 20:13:56.580261 31538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:56.580504 31343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 20:13:56.580574 31343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/wal
uuid: "08260900a88b4eb59a21b4a0071e85f3"
format_stamp: "Formatted at 2026-06-20 20:13:56 on dist-test-slave-pflr"
server_key: "df2abd74d95524ee0acd1222bb7d8da0"
server_key_iv: "ccf888c02e9ab4ad7d9b62bb6941a3f7"
server_key_version: "encryptionkey@0"
I20260620 20:13:56.580708 31343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:56.597182 31343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:56.599268 31343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:56.604187 31343 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.188:45015
I20260620 20:13:56.604220 31590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.188:45015 every 8 connection(s)
I20260620 20:13:56.604790 31343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/master-2/data/info.pb
I20260620 20:13:56.607615 31591 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:13:56.608919 31591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: Bootstrap starting.
I20260620 20:13:56.620155 31591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:13:56.620443 31591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: Bootstrap complete.
I20260620 20:13:56.621037 31591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } }
I20260620 20:13:56.621176 31591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08260900a88b4eb59a21b4a0071e85f3, State: Initialized, Role: LEARNER
I20260620 20:13:56.621292 31591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } }
I20260620 20:13:56.621675 31594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } } }
I20260620 20:13:56.621773 31591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:13:56.621878 31594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:13:56.625300 31605 catalog_manager.cc:1259] Loaded cluster ID: 4ba2f43188c24306b8eda927b1c49d9b
I20260620 20:13:56.625355 31605 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: loading cluster ID for follower catalog manager: success
I20260620 20:13:56.627393 31605 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: acquiring CA information for follower catalog manager: success
I20260620 20:13:56.628743 31605 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 20:13:56.644131 31516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:56.644357 31516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:56.644426 31516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:56.648386 31516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:56.648442 31516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:56.648538 31516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:13:56.653753 31516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab
--principal=oryx/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:41279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=39037
--tserver_master_addrs=127.24.20.190:44703,127.24.20.189:39307,127.24.20.188:45015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:56.654799 31516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:56.655817 31516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:56.666543 31611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:56.666617 31610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:56.666646 31613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:56.666625 31516 server_base.cc:1034] running on GCE node
I20260620 20:13:56.667124 31516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:56.667701 31516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:56.668884 31516 hybrid_clock.cc:648] HybridClock initialized: now 1781986436668857 us; error 45 us; skew 500 ppm
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:56.671397 31516 init.cc:377] Logged in from keytab as oryx/127.24.20.129@KRBTEST.COM (short username oryx)
I20260620 20:13:56.672363 31516 webserver.cc:460] Webserver started at http://127.24.20.129:39037/ using document root <none> and password file <none>
I20260620 20:13:56.673018 31516 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:56.673125 31516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:56.676697 31516 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 20:13:56.678535 31620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:56.679669 31516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:56.679797 31516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "d1a7fa0094974f778502f3d8de7c5835"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "ff6d002f19e0abf3530d73e14e09331b"
server_key_iv: "0b6cc35bc7837c23999963b0dc5896a3"
server_key_version: "encryptionkey@0"
I20260620 20:13:56.680303 31516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:56.681371 31096 heartbeater.cc:503] Master 127.24.20.190:44703 was elected leader, sending a full tablet report...
I20260620 20:13:56.691747 31516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:56.694494 31516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:56.694689 31516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:56.695215 31516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:56.695986 31516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:56.696053 31516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:56.696110 31516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:56.696158 31516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:56.704914 31516 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:41279
I20260620 20:13:56.704937 31733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:41279 every 8 connection(s)
I20260620 20:13:56.705960 31516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:13:56.709396 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 31516
I20260620 20:13:56.709586 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 30967
I20260620 20:13:56.717080 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:44783
--local_ip_for_outbound_sockets=127.24.20.130
--tserver_master_addrs=127.24.20.190:44703
--webserver_port=33667
--webserver_interface=127.24.20.130
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab
--principal=oryx/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.20.190:44703,127.24.20.189:39307
--tserver_master_addrs=127.24.20.190:44703,127.24.20.189:39307,127.24.20.188:45015 with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.189@KRBTEST.COM
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.188@KRBTEST.COM
I20260620 20:13:56.723829 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.709233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:54123 (local address 127.24.20.188:45015)
0620 20:13:56.709472 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.709485 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.709501 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:13:56.710478 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.710482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.710533 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.710613 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.713235 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.714311 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.715476 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.715799 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.715897 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.719604 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.719610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.719618 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.719652 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.721705 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.722173 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.722177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.722179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.722263 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.722614 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.722618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.722622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.722851 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.722928 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.723516 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.723656 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
I20260620 20:13:56.723963 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.707814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:45917 (local address 127.24.20.190:44703)
0620 20:13:56.707958 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.707964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.708739 (+ 775us) server_negotiation.cc:407] Connection header received
0620 20:13:56.710231 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.710234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.710297 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.710376 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.712421 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.713401 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.715007 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.715440 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.715538 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.719190 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.719194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.719196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.719227 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.721077 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.721714 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.721719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.721723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.721840 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.722332 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.722338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.722343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.722672 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.722759 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.723573 (+ 814us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.723710 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:13:56.724210 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.707741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:44487 (local address 127.24.20.189:39307)
0620 20:13:56.707877 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.707882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.708731 (+ 849us) server_negotiation.cc:407] Connection header received
0620 20:13:56.710403 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.710409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.710478 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.710563 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.712546 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.713243 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.714967 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.715304 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.715379 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.719966 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.719971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.719975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.720006 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.722541 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.722972 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.722976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.722980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.723056 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.723361 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.723365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.723366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.723567 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.723633 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.723951 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.724062 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:13:56.725379 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.708134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39307 (local address 127.24.20.129:44487)
0620 20:13:56.708569 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.708645 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.710291 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.710722 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.710728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.711551 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.712307 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.712318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.713396 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.713399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.714839 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.714849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.715567 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.715570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.715764 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.716754 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.716767 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.719464 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.722648 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.722651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.722652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.722865 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.723175 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.723179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.723181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.723259 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.723705 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.723708 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.723782 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.724395 (+ 613us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.724507 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":407,"spinlock_wait_cycles":9088,"thread_start_us":130,"threads_started":1}
I20260620 20:13:56.725392 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.708133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.129:45917)
0620 20:13:56.708570 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.708641 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.709984 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.710707 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.710715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.711492 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.712251 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.712272 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.713549 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.713555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.714873 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.714886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.715702 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.715708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.715830 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.716461 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.716486 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.718981 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.721210 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.721218 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.721236 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.721573 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.722005 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.722011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.722014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.722202 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.722892 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.722901 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.723277 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.724154 (+ 877us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.724652 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":434,"spinlock_wait_cycles":53504,"thread_start_us":145,"threads_started":1}
I20260620 20:13:56.726118 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.709091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:45015 (local address 127.24.20.129:54123)
0620 20:13:56.709336 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.709350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.710374 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.710721 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.710723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.712027 (+ 1304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.712420 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.712430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.714438 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.714441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.715351 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.715359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.716044 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.716048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.716170 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.716633 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.716649 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.719464 (+ 2815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.721806 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.721812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.721815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.722057 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.722383 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.722388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.722392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.722506 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.722994 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.722997 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.723258 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.723986 (+ 728us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.724302 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":220,"thread_start_us":83,"threads_started":1}
I20260620 20:13:56.726573 31736 heartbeater.cc:347] Connected to a master server at 127.24.20.189:39307
I20260620 20:13:56.726786 31735 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44703
I20260620 20:13:56.726893 31735 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:56.726975 31734 heartbeater.cc:347] Connected to a master server at 127.24.20.188:45015
I20260620 20:13:56.727048 31734 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:56.726811 31736 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:56.727442 31734 heartbeater.cc:511] Master 127.24.20.188:45015 requested a full tablet report, sending...
I20260620 20:13:56.727488 31736 heartbeater.cc:511] Master 127.24.20.189:39307 requested a full tablet report, sending...
I20260620 20:13:56.727459 31735 heartbeater.cc:511] Master 127.24.20.190:44703 requested a full tablet report, sending...
I20260620 20:13:56.728772 30771 ts_manager.cc:194] Re-registered known tserver with Master: d1a7fa0094974f778502f3d8de7c5835 (127.24.20.129:41279)
I20260620 20:13:56.729187 31547 ts_manager.cc:194] Registered new tserver with Master: d1a7fa0094974f778502f3d8de7c5835 (127.24.20.129:41279)
I20260620 20:13:56.729215 31371 ts_manager.cc:194] Registered new tserver with Master: d1a7fa0094974f778502f3d8de7c5835 (127.24.20.129:41279)
I20260620 20:13:56.729466 30771 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:45917
W20260620 20:13:56.832762 31742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:56.832996 31742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:56.833045 31742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:56.836949 31742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:56.837002 31742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:56.837072 31742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:13:56.842249 31742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab
--principal=oryx/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:44783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=33667
--tserver_master_addrs=127.24.20.190:44703,127.24.20.189:39307,127.24.20.188:45015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:56.843307 31742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:56.844323 31742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:13:56.853654 31747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:56.853667 31748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:56.853773 31750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:56.853976 31742 server_base.cc:1034] running on GCE node
I20260620 20:13:56.854287 31742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:56.854846 31742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:56.856024 31742 hybrid_clock.cc:648] HybridClock initialized: now 1781986436856009 us; error 32 us; skew 500 ppm
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:56.858557 31742 init.cc:377] Logged in from keytab as oryx/127.24.20.130@KRBTEST.COM (short username oryx)
I20260620 20:13:56.859416 31742 webserver.cc:460] Webserver started at http://127.24.20.130:33667/ using document root <none> and password file <none>
I20260620 20:13:56.859936 31742 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:56.860038 31742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:56.863914 31742 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:13:56.865995 31757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:56.867108 31742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:56.867247 31742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "bb509074a785475ea0930e20cc2a2107"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "88050ab90711c09a7a730737341640c0"
server_key_iv: "b76ac91b9b5d0d72c582162e9bf4432f"
server_key_version: "encryptionkey@0"
I20260620 20:13:56.867749 31742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:56.879323 31742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:56.882316 31742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:56.882531 31742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:56.883139 31742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:56.883926 31742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:56.883993 31742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:56.884055 31742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:56.884106 31742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:56.892359 31742 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:44783
I20260620 20:13:56.892390 31870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:44783 every 8 connection(s)
I20260620 20:13:56.893375 31742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:13:56.893708 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 31742
I20260620 20:13:56.893903 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 31101
I20260620 20:13:56.900928 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:44259
--local_ip_for_outbound_sockets=127.24.20.131
--tserver_master_addrs=127.24.20.190:44703
--webserver_port=40439
--webserver_interface=127.24.20.131
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab
--principal=oryx/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.20.190:44703,127.24.20.189:39307
--tserver_master_addrs=127.24.20.190:44703,127.24.20.189:39307,127.24.20.188:45015 with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.188@KRBTEST.COM
Jun 20 20:13:56 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986436, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.189@KRBTEST.COM
I20260620 20:13:56.911798 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.895371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:40103 (local address 127.24.20.189:39307)
0620 20:13:56.895540 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.895547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.896663 (+ 1116us) server_negotiation.cc:407] Connection header received
0620 20:13:56.897834 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.897838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.897890 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.897945 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.899829 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.900664 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.902626 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.902992 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.903068 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.907536 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.907540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.907543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.907570 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.909629 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.910197 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.910204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.910209 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.910295 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.910633 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.910637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.910638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.910852 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.910915 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.911493 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.911614 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:13:56.911966 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.897277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:60671 (local address 127.24.20.188:45015)
0620 20:13:56.897422 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.897429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.897710 (+ 281us) server_negotiation.cc:407] Connection header received
0620 20:13:56.897886 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.897893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.897955 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.898011 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.899830 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.900678 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.902636 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.902994 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.903068 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.907536 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.907540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.907543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.907570 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.909743 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.910228 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.910231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.910233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.910307 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.910643 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.910646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.910648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.910842 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.910915 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.911493 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.911795 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:13:56.912011 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.897544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:52585 (local address 127.24.20.190:44703)
0620 20:13:56.897669 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:56.897674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:56.898603 (+ 929us) server_negotiation.cc:407] Connection header received
0620 20:13:56.898774 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:56.898780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:56.898837 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:56.898892 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:56.899825 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.900695 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.902636 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.903007 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.903089 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.906603 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:56.906609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:56.906613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:56.906648 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:56.909055 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.909788 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.909793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.909798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.909897 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:56.910354 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:56.910360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:56.910363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:56.910641 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:56.910728 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:56.911533 (+ 805us) server_negotiation.cc:299] Negotiation successful
0620 20:13:56.911785 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:13:56.913470 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.897245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.130:52585)
0620 20:13:56.898503 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.898530 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.898669 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.899042 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.899047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.899253 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.899660 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.899681 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.900958 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.900961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.902484 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.902498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.903267 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.903272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.903517 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.904548 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.904568 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.906379 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.909201 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.909208 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.909229 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.909653 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.910025 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.910029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.910032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.910227 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.910821 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.910831 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.911205 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.912289 (+ 1084us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.912463 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"mutex_wait_us":222,"spinlock_wait_cycles":15232,"thread_start_us":73,"threads_started":1}
I20260620 20:13:56.913484 31877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.895613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:45015 (local address 127.24.20.130:60671)
0620 20:13:56.897618 (+ 2005us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.897634 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.897777 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.898274 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.898280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.898815 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.899667 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.899681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.900844 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.900847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.902490 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.902500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.903260 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.903263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.903520 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.904379 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.904390 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.907381 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.909848 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.909852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.909854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.910126 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.910414 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.910417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.910419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.910528 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.910989 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.910992 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.911201 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.912659 (+ 1458us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.912793 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1900,"mutex_wait_us":1769,"spinlock_wait_cycles":15232,"thread_start_us":114,"threads_started":1}
I20260620 20:13:56.913486 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:56.895613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39307 (local address 127.24.20.130:40103)
0620 20:13:56.896492 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:56.896538 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:56.897604 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:56.898272 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:56.898277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:56.898815 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:56.899663 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.899681 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.900844 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.900847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.902490 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:56.902500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:56.903260 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:56.903264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:56.903518 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:56.904316 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:56.904357 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:56.907381 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:56.909759 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.909763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.909767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.910069 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.910414 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:56.910417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:56.910419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:56.910534 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:56.910989 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:56.910992 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:56.911201 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 20:13:56.911959 (+ 758us) client_negotiation.cc:240] Negotiation successful
0620 20:13:56.912368 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":393,"spinlock_wait_cycles":24576,"thread_start_us":546,"threads_started":1}
I20260620 20:13:56.914752 31873 heartbeater.cc:347] Connected to a master server at 127.24.20.189:39307
I20260620 20:13:56.914991 31872 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44703
I20260620 20:13:56.915021 31873 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:56.915052 31872 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:56.914767 31871 heartbeater.cc:347] Connected to a master server at 127.24.20.188:45015
I20260620 20:13:56.915442 31871 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:56.915575 31871 heartbeater.cc:511] Master 127.24.20.188:45015 requested a full tablet report, sending...
I20260620 20:13:56.915575 31872 heartbeater.cc:511] Master 127.24.20.190:44703 requested a full tablet report, sending...
I20260620 20:13:56.916169 31873 heartbeater.cc:511] Master 127.24.20.189:39307 requested a full tablet report, sending...
I20260620 20:13:56.916824 31371 ts_manager.cc:194] Registered new tserver with Master: bb509074a785475ea0930e20cc2a2107 (127.24.20.130:44783)
I20260620 20:13:56.916818 30771 ts_manager.cc:194] Re-registered known tserver with Master: bb509074a785475ea0930e20cc2a2107 (127.24.20.130:44783)
I20260620 20:13:56.916937 31547 ts_manager.cc:194] Registered new tserver with Master: bb509074a785475ea0930e20cc2a2107 (127.24.20.130:44783)
I20260620 20:13:56.917544 30771 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:52585
W20260620 20:13:57.014935 31878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:13:57.015308 31878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:13:57.015354 31878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:13:57.019512 31878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:13:57.019572 31878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:13:57.019629 31878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:13:57.024927 31878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab
--principal=oryx/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:44259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=40439
--tserver_master_addrs=127.24.20.190:44703,127.24.20.189:39307,127.24.20.188:45015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:13:57.025892 31878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:13:57.027067 31878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:13:57.036463 31878 server_base.cc:1034] running on GCE node
W20260620 20:13:57.036418 31883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:57.036478 31884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:13:57.036413 31886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:13:57.037117 31878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:13:57.037688 31878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:13:57.038841 31878 hybrid_clock.cc:648] HybridClock initialized: now 1781986437038823 us; error 50 us; skew 500 ppm
Jun 20 20:13:57 dist-test-slave-pflr krb5kdc[30726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986437, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:13:57.041712 31878 init.cc:377] Logged in from keytab as oryx/127.24.20.131@KRBTEST.COM (short username oryx)
I20260620 20:13:57.042639 31878 webserver.cc:460] Webserver started at http://127.24.20.131:40439/ using document root <none> and password file <none>
I20260620 20:13:57.043200 31878 fs_manager.cc:359] Metadata directory not provided
I20260620 20:13:57.043275 31878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:13:57.046936 31878 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 20:13:57.049165 31893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:57.050412 31878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:13:57.050652 31878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "b6deecfe720547eca4dd1bd60cd96add"
format_stamp: "Formatted at 2026-06-20 20:13:55 on dist-test-slave-pflr"
server_key: "7eb15c318315bc3f5f5b8b047f5ae39e"
server_key_iv: "4d263d10f651c618070843efb5530dd3"
server_key_version: "encryptionkey@0"
I20260620 20:13:57.051170 31878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:13:57.059998 31878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:13:57.062744 31878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:13:57.062927 31878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:13:57.063470 31878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:13:57.064248 31878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:13:57.064292 31878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:57.064328 31878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:13:57.064388 31878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:13:57.073191 31878 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:44259
I20260620 20:13:57.073233 32006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:44259 every 8 connection(s)
I20260620 20:13:57.074159 31878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:13:57.077838 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 31878
Jun 20 20:13:57 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986437, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.188@KRBTEST.COM
I20260620 20:13:57.092085 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.085296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:45015 (local address 127.0.0.1:43022)
0620 20:13:57.085578 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:57.085616 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:57.085744 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:57.088028 (+ 2284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:57.088033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:57.088054 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:57.088148 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.088155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.089185 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.089189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.090471 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.090482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.091586 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.091593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.091714 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.091758 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 20:13:57.091814 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 20:13:57.091907 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
I20260620 20:13:57.092777 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.077312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:45015 (local address 127.24.20.131:56463)
0620 20:13:57.077610 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:57.077621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:57.078142 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:57.078597 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:57.078607 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:57.079745 (+ 1138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:57.079889 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.079899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.081197 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.081202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.082522 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.082533 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.083550 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.083556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.083839 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.084506 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:57.084525 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:57.086604 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:57.089144 (+ 2540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:57.089154 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:57.089170 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:57.089800 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:57.090094 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:57.090097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:57.090099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:57.090280 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:57.090740 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:57.090749 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:57.091053 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 20:13:57.091278 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 20:13:57.091649 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":58,"thread_start_us":77,"threads_started":1}
I20260620 20:13:57.092881 32021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.086095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.24.20.188:45015)
0620 20:13:57.087765 (+ 1670us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.087777 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.087787 (+ 10us) server_negotiation.cc:407] Connection header received
0620 20:13:57.087824 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.087827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.087879 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.088244 (+ 365us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:57.088329 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.089061 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.090628 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.091378 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.092375 (+ 997us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.092553 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.092652 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.092736 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1609,"thread_start_us":87,"threads_started":1}
I20260620 20:13:57.093469 31565 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 20:13:57.093793 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.077415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:56463 (local address 127.24.20.188:45015)
0620 20:13:57.077551 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.077556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.077697 (+ 141us) server_negotiation.cc:407] Connection header received
0620 20:13:57.078258 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.078264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.078321 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.078384 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:57.080061 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.080993 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.082661 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.083027 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.083121 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.086756 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:57.086760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:57.086763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:57.086789 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:57.088789 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.089912 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.089916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.089918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.089999 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.090377 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.090380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.090382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.090593 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:57.090684 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.093499 (+ 2815us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.093622 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:13:57.094003 31594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0bf847e90b54356bcdd594d09491430. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } } }
I20260620 20:13:57.094151 31594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:13:57.094568 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.081081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.24.20.189:39307)
0620 20:13:57.081983 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.082007 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.082277 (+ 270us) server_negotiation.cc:407] Connection header received
0620 20:13:57.082586 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.082589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.082675 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.082782 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:57.085717 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.086496 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.087642 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.088445 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.088536 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.090346 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:57.090350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:57.090353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:57.090376 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:57.092436 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.092932 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.092938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.092942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.093032 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.093341 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.093344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.093345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.093556 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:57.093683 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.094265 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.094405 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":486,"server-negotiator.queue_time_us":809,"thread_start_us":78,"threads_started":1}
I20260620 20:13:57.095443 32007 heartbeater.cc:347] Connected to a master server at 127.24.20.188:45015
I20260620 20:13:57.095687 32007 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:57.096091 32023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.084020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.24.20.190:44703)
0620 20:13:57.084414 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.084438 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.086877 (+ 2439us) server_negotiation.cc:407] Connection header received
0620 20:13:57.087075 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.087079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.087132 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.087192 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:57.088243 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.089099 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.090270 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.090594 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.090670 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.092272 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:57.092275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:57.092278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:57.092302 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:57.094153 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.094640 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.094645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.094647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.094746 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.095039 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.095043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.095045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.095280 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:57.095396 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.095754 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.095879 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":147,"threads_started":1}
I20260620 20:13:57.096235 32007 heartbeater.cc:511] Master 127.24.20.188:45015 requested a full tablet report, sending...
I20260620 20:13:57.097410 31547 ts_manager.cc:194] Registered new tserver with Master: b6deecfe720547eca4dd1bd60cd96add (127.24.20.131:44259)
Jun 20 20:13:57 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986437, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.189@KRBTEST.COM
Jun 20 20:13:57 dist-test-slave-pflr krb5kdc[30726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986437, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
I20260620 20:13:57.102568 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.075945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:44593 (local address 127.24.20.189:39307)
0620 20:13:57.076098 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.076102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.077349 (+ 1247us) server_negotiation.cc:407] Connection header received
0620 20:13:57.078163 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.078166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.078220 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.078297 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:57.082642 (+ 4345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.086449 (+ 3807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.087642 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.087996 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.088066 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.098418 (+ 10352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:57.098433 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:57.098438 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:57.098460 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:57.100862 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.101322 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.101325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.101327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.101401 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.101686 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.101689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.101691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.101873 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:57.101938 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.102253 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.102358 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":61}
I20260620 20:13:57.099577 30771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 20:13:57.103489 32012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.076358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39307 (local address 127.24.20.131:44593)
0620 20:13:57.077266 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:57.077279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:57.077956 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:57.078597 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:57.078607 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:57.079184 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:57.082515 (+ 3331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.082526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.086593 (+ 4067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.086596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.087500 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.087511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.089016 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.089019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.089120 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.096677 (+ 7557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:57.096690 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:57.098274 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:57.100980 (+ 2706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:57.100984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:57.100985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:57.101219 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:57.101515 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:57.101517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:57.101519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:57.101597 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:57.102003 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:57.102006 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:57.102088 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 20:13:57.102738 (+ 650us) client_negotiation.cc:240] Negotiation successful
0620 20:13:57.102877 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":2910,"spinlock_wait_cycles":11648,"thread_start_us":185,"threads_started":1}
I20260620 20:13:57.103753 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.076654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:37233 (local address 127.24.20.190:44703)
0620 20:13:57.076775 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.076779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.077011 (+ 232us) server_negotiation.cc:407] Connection header received
0620 20:13:57.078875 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.078880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.078948 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.079012 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:57.080044 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.080829 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.095892 (+ 15063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.096214 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.096296 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.100119 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:57.100124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:57.100129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:57.100157 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:57.101875 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.102386 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.102390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.102392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.102498 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.102824 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.102827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.102829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.103030 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:57.103128 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.103475 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.103589 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:13:57.104041 32009 heartbeater.cc:347] Connected to a master server at 127.24.20.189:39307
I20260620 20:13:57.104135 32009 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:57.104295 32011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.076358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.131:37233)
0620 20:13:57.076847 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:57.076896 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:57.077957 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:57.079167 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:57.079170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:57.079380 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:13:57.079857 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.079877 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.081013 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.081016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.095767 (+ 14751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.095776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.096445 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.096448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.096549 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.098472 (+ 1923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:13:57.098486 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:13:57.099961 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:13:57.102002 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:57.102008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:57.102011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:57.102270 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:57.102619 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:13:57.102621 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:13:57.102623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:13:57.102728 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:13:57.103183 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:13:57.103186 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:13:57.103273 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 20:13:57.103888 (+ 615us) client_negotiation.cc:240] Negotiation successful
0620 20:13:57.104034 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":13298,"thread_start_us":165,"threads_started":1}
I20260620 20:13:57.104311 32009 heartbeater.cc:511] Master 127.24.20.189:39307 requested a full tablet report, sending...
W20260620 20:13:57.104701 30771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:57.105005 32008 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44703
I20260620 20:13:57.105082 32008 heartbeater.cc:464] Registering TS with master...
I20260620 20:13:57.105242 32008 heartbeater.cc:511] Master 127.24.20.190:44703 requested a full tablet report, sending...
I20260620 20:13:57.105422 31371 ts_manager.cc:194] Registered new tserver with Master: b6deecfe720547eca4dd1bd60cd96add (127.24.20.131:44259)
I20260620 20:13:57.105819 30770 ts_manager.cc:194] Re-registered known tserver with Master: b6deecfe720547eca4dd1bd60cd96add (127.24.20.131:44259)
I20260620 20:13:57.106328 30770 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:37233
I20260620 20:13:57.106809 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.083402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43006 (local address 127.24.20.188:45015)
0620 20:13:57.086315 (+ 2913us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.086319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.095483 (+ 9164us) server_negotiation.cc:407] Connection header received
0620 20:13:57.095521 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.095524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.095564 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.095645 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:57.096549 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.097263 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.098729 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.099197 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.099299 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.100957 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:57.100962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:57.100965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:57.100993 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:57.103261 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.104206 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.104210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.104212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.104296 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.104588 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.104615 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.104616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.104823 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:57.104924 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.105616 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.106659 (+ 1043us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2841,"thread_start_us":212,"threads_started":1}
I20260620 20:13:57.108249 32024 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: NON_VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: true } } }
I20260620 20:13:57.108376 32024 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:57.111584 32024 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430: attempting to promote NON_VOTER 08260900a88b4eb59a21b4a0071e85f3 to VOTER
I20260620 20:13:57.111929 32024 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } }
I20260620 20:13:57.113090 31565 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 1 LEARNER]: Refusing update from remote peer c0bf847e90b54356bcdd594d09491430: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 20:13:57.113639 32029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 20:13:57.115240 32024 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 08260900a88b4eb59a21b4a0071e85f3 (127.24.20.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } } }
I20260620 20:13:57.115845 31565 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 08260900a88b4eb59a21b4a0071e85f3 (127.24.20.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } } }
I20260620 20:13:57.116683 31381 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 FOLLOWER]: Refusing update from remote peer c0bf847e90b54356bcdd594d09491430: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 20:13:57.116842 32024 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } } }
I20260620 20:13:57.116940 32024 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:57.117069 32024 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } } }
I20260620 20:13:57.117147 32024 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:57.117146 32030 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 20:13:57.117941 31594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } } }
I20260620 20:13:57.118054 31594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:13:57.118322 31381 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 08260900a88b4eb59a21b4a0071e85f3 (127.24.20.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } } }
I20260620 20:13:57.120015 32038 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } } }
I20260620 20:13:57.120415 32024 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0bf847e90b54356bcdd594d09491430" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } } }
I20260620 20:13:57.120540 32024 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:57.120137 32038 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:13:57.126839 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.121941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:44783 (local address 127.0.0.1:60076)
0620 20:13:57.122768 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:57.122783 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:57.122936 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:57.123509 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:57.123512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:57.123531 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:57.123619 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.123625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.124500 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.124505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.125672 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.125683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.126431 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.126435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.126517 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.126540 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:13:57.126629 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 20:13:57.126699 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260620 20:13:57.126865 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.122096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.24.20.130:44783)
0620 20:13:57.122426 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.122446 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.122861 (+ 415us) server_negotiation.cc:407] Connection header received
0620 20:13:57.123139 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.123145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.123370 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.123522 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:57.123709 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.124389 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.125773 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.126303 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.126424 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.126561 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.126655 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.126743 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":111,"threads_started":1}
I20260620 20:13:57.126982 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.121385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:44259 (local address 127.0.0.1:37866)
0620 20:13:57.121483 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:57.121495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:57.121562 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:57.122400 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:57.122403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:57.122434 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:57.122543 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.122550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.124166 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.124169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.125354 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.125364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.126619 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.126623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.126709 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.126729 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:13:57.126795 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 20:13:57.126861 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":26}
I20260620 20:13:57.127677 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.121556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37866 (local address 127.24.20.131:44259)
0620 20:13:57.121817 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.121827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.121846 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:13:57.121990 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.121997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.122207 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.122441 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:57.122972 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.124053 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.125509 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.126308 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.126404 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.126560 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.127463 (+ 903us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.127542 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
I20260620 20:13:57.128837 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.122547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:41279 (local address 127.0.0.1:54574)
0620 20:13:57.123268 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:57.123280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:57.123358 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:57.124771 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:57.124774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:57.124792 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:57.124870 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.124879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.126165 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.126168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.127275 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.127285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.128453 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.128457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.128541 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.128562 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:13:57.128656 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 20:13:57.128722 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":86,"threads_started":1}
I20260620 20:13:57.129207 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.122682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.24.20.129:41279)
0620 20:13:57.124330 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.124342 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.124362 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:13:57.124430 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.124435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.124639 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.124783 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:57.124981 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.126036 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.127389 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.128152 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.128271 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.128388 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.128994 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.129083 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1550,"thread_start_us":93,"threads_started":1}
I20260620 20:13:57.129825 31805 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd4bffef80542618e90e5cec6d7891d (DEFAULT_TABLE table=test-table [id=44ffe130d89c4bec9acd82e2156e9463]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:57.130191 31941 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd4bffef80542618e90e5cec6d7891d (DEFAULT_TABLE table=test-table [id=44ffe130d89c4bec9acd82e2156e9463]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:57.130672 31805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd4bffef80542618e90e5cec6d7891d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:57.131052 31941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd4bffef80542618e90e5cec6d7891d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:57.134815 31668 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd4bffef80542618e90e5cec6d7891d (DEFAULT_TABLE table=test-table [id=44ffe130d89c4bec9acd82e2156e9463]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:57.135799 31668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd4bffef80542618e90e5cec6d7891d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:57.138129 32044 tablet_bootstrap.cc:492] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107: Bootstrap starting.
I20260620 20:13:57.139081 32045 tablet_bootstrap.cc:492] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add: Bootstrap starting.
I20260620 20:13:57.140029 32044 tablet_bootstrap.cc:654] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:57.140892 32044 log.cc:824] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:57.141273 32045 tablet_bootstrap.cc:654] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:57.142064 32045 log.cc:824] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:57.143005 32044 tablet_bootstrap.cc:492] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107: No bootstrap required, opened a new log
I20260620 20:13:57.143290 32044 ts_tablet_manager.cc:1397] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:13:57.143381 32048 tablet_bootstrap.cc:492] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835: Bootstrap starting.
I20260620 20:13:57.143910 32045 tablet_bootstrap.cc:492] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add: No bootstrap required, opened a new log
I20260620 20:13:57.144107 32045 ts_tablet_manager.cc:1397] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:13:57.145470 32048 tablet_bootstrap.cc:654] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:57.146211 32048 log.cc:824] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835: Log is configured to *not* fsync() on all Append() calls
I20260620 20:13:57.147643 32045 raft_consensus.cc:348] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } } peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } }
I20260620 20:13:57.147861 32045 raft_consensus.cc:374] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:57.147912 32045 raft_consensus.cc:729] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6deecfe720547eca4dd1bd60cd96add, State: Initialized, Role: FOLLOWER
I20260620 20:13:57.147783 32044 raft_consensus.cc:348] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } } peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } }
I20260620 20:13:57.148038 32044 raft_consensus.cc:374] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:57.148103 32044 raft_consensus.cc:729] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb509074a785475ea0930e20cc2a2107, State: Initialized, Role: FOLLOWER
I20260620 20:13:57.148157 32048 tablet_bootstrap.cc:492] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835: No bootstrap required, opened a new log
I20260620 20:13:57.148310 32048 ts_tablet_manager.cc:1397] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:13:57.148411 32045 consensus_queue.cc:260] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } } peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } }
I20260620 20:13:57.148506 32044 consensus_queue.cc:260] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } } peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } }
I20260620 20:13:57.149391 32045 ts_tablet_manager.cc:1428] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:13:57.149417 32044 ts_tablet_manager.cc:1428] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:13:57.149660 32008 heartbeater.cc:503] Master 127.24.20.190:44703 was elected leader, sending a full tablet report...
I20260620 20:13:57.151655 31872 heartbeater.cc:503] Master 127.24.20.190:44703 was elected leader, sending a full tablet report...
I20260620 20:13:57.153960 32048 raft_consensus.cc:348] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } } peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } }
I20260620 20:13:57.154204 32048 raft_consensus.cc:374] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:57.154266 32048 raft_consensus.cc:729] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a7fa0094974f778502f3d8de7c5835, State: Initialized, Role: FOLLOWER
I20260620 20:13:57.154722 32048 consensus_queue.cc:260] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } } peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } }
I20260620 20:13:57.155476 31735 heartbeater.cc:503] Master 127.24.20.190:44703 was elected leader, sending a full tablet report...
I20260620 20:13:57.155723 32048 ts_tablet_manager.cc:1428] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260620 20:13:57.207513 31737 tablet.cc:2369] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:13:57.251060 32052 raft_consensus.cc:482] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:57.251284 32052 raft_consensus.cc:504] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } } peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } }
I20260620 20:13:57.252455 32052 leader_election.cc:288] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb509074a785475ea0930e20cc2a2107 (127.24.20.130:44783), b6deecfe720547eca4dd1bd60cd96add (127.24.20.131:44259)
I20260620 20:13:57.256402 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.252793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:37261 (local address 127.24.20.130:44783)
0620 20:13:57.252916 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.252921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.253145 (+ 224us) server_negotiation.cc:407] Connection header received
0620 20:13:57.253296 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.253298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.253343 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.253422 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:57.253649 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.254332 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.255504 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.255878 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.255997 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.256051 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.256212 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.256277 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 20:13:57.256457 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.252730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:44783 (local address 127.24.20.129:37261)
0620 20:13:57.253059 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:57.253071 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:57.253196 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:57.253433 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:57.253436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:57.253452 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:57.253553 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.253558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.254426 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.254432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.255382 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.255392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.255988 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.255999 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.256089 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.256111 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:13:57.256198 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 20:13:57.256267 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":47,"thread_start_us":86,"threads_started":1}
I20260620 20:13:57.256453 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.252959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:52855 (local address 127.24.20.131:44259)
0620 20:13:57.253092 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.253097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.253244 (+ 147us) server_negotiation.cc:407] Connection header received
0620 20:13:57.253387 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.253391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.253436 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.253508 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:57.253759 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.254322 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.255538 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.255923 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.256044 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.256091 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.256265 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.256327 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:13:57.256696 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.252736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:44259 (local address 127.24.20.129:52855)
0620 20:13:57.253162 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:57.253173 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:57.253282 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:57.253530 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:57.253532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:57.253548 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:57.253655 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.253660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.254419 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.254425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.255422 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:57.255431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.256036 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.256040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:57.256144 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.256164 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:13:57.256250 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 20:13:57.256316 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":80,"thread_start_us":71,"threads_started":1}
I20260620 20:13:57.257159 31825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd4bffef80542618e90e5cec6d7891d" candidate_uuid: "d1a7fa0094974f778502f3d8de7c5835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb509074a785475ea0930e20cc2a2107" is_pre_election: true
I20260620 20:13:57.257321 31961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd4bffef80542618e90e5cec6d7891d" candidate_uuid: "d1a7fa0094974f778502f3d8de7c5835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6deecfe720547eca4dd1bd60cd96add" is_pre_election: true
I20260620 20:13:57.257462 31825 raft_consensus.cc:2454] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a7fa0094974f778502f3d8de7c5835 in term 0.
I20260620 20:13:57.257611 31961 raft_consensus.cc:2454] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1a7fa0094974f778502f3d8de7c5835 in term 0.
I20260620 20:13:57.257917 31622 leader_election.cc:302] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb509074a785475ea0930e20cc2a2107, d1a7fa0094974f778502f3d8de7c5835; no voters:
I20260620 20:13:57.258169 32052 raft_consensus.cc:2781] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:57.258255 32052 raft_consensus.cc:482] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:57.258313 32052 raft_consensus.cc:3037] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:57.259274 32052 raft_consensus.cc:504] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } } peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } }
I20260620 20:13:57.259653 32052 leader_election.cc:288] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [CANDIDATE]: Term 1 election: Requested vote from peers bb509074a785475ea0930e20cc2a2107 (127.24.20.130:44783), b6deecfe720547eca4dd1bd60cd96add (127.24.20.131:44259)
I20260620 20:13:57.259943 31825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd4bffef80542618e90e5cec6d7891d" candidate_uuid: "d1a7fa0094974f778502f3d8de7c5835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb509074a785475ea0930e20cc2a2107"
I20260620 20:13:57.260066 31825 raft_consensus.cc:3037] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:57.261160 31825 raft_consensus.cc:2454] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a7fa0094974f778502f3d8de7c5835 in term 1.
I20260620 20:13:57.261407 31622 leader_election.cc:302] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb509074a785475ea0930e20cc2a2107, d1a7fa0094974f778502f3d8de7c5835; no voters:
I20260620 20:13:57.261559 32052 raft_consensus.cc:2781] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:57.261772 32052 raft_consensus.cc:686] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [term 1 LEADER]: Becoming Leader. State: Replica: d1a7fa0094974f778502f3d8de7c5835, State: Running, Role: LEADER
I20260620 20:13:57.262342 32052 consensus_queue.cc:237] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } } peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } }
I20260620 20:13:57.265715 30770 catalog_manager.cc:5495] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1a7fa0094974f778502f3d8de7c5835 (127.24.20.129). New cstate: current_term: 1 leader_uuid: "d1a7fa0094974f778502f3d8de7c5835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } health_report { overall_health: UNKNOWN } } }
I20260620 20:13:57.266626 31961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd4bffef80542618e90e5cec6d7891d" candidate_uuid: "d1a7fa0094974f778502f3d8de7c5835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6deecfe720547eca4dd1bd60cd96add"
I20260620 20:13:57.266744 31961 raft_consensus.cc:3037] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:57.267974 31961 raft_consensus.cc:2454] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1a7fa0094974f778502f3d8de7c5835 in term 1.
I20260620 20:13:57.314184 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.310953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.24.20.129:41279)
0620 20:13:57.311080 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.311083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.311096 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:13:57.311220 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.311223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.311274 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.311358 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:57.311552 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.312036 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.312889 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.313095 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.313176 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.313394 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:57.313825 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:57.313910 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.313970 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.314018 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:13:57.319877 31825 raft_consensus.cc:1261] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107 [term 1 FOLLOWER]: Refusing update from remote peer d1a7fa0094974f778502f3d8de7c5835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:57.319937 31961 raft_consensus.cc:1261] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add [term 1 FOLLOWER]: Refusing update from remote peer d1a7fa0094974f778502f3d8de7c5835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:57.320637 32052 consensus_queue.cc:1035] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:57.320849 32055 consensus_queue.cc:1035] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:13:57.325920 32010 tablet.cc:2369] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:13:57.331146 32058 mvcc.cc:205] Tried to move back new op lower bound from 7299016447255371776 to 7299016447028989952. Current Snapshot: MvccSnapshot[applied={T|T < 7299016447255371776 or (T in {7299016447255371776})}]
I20260620 20:13:57.332912 32060 mvcc.cc:205] Tried to move back new op lower bound from 7299016447255371776 to 7299016447028989952. Current Snapshot: MvccSnapshot[applied={T|T < 7299016447255371776}]
I20260620 20:13:57.333987 32059 mvcc.cc:205] Tried to move back new op lower bound from 7299016447255371776 to 7299016447028989952. Current Snapshot: MvccSnapshot[applied={T|T < 7299016447255371776 or (T in {7299016447255371776})}]
I20260620 20:13:57.338333 30770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:57.338460 30770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:57.342299 30770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0bf847e90b54356bcdd594d09491430: Sending DeleteTablet for 3 replicas of tablet 2fd4bffef80542618e90e5cec6d7891d
I20260620 20:13:57.343152 31941 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fd4bffef80542618e90e5cec6d7891d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:57 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:37866
I20260620 20:13:57.343191 31805 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fd4bffef80542618e90e5cec6d7891d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:57 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:60076
I20260620 20:13:57.343534 31668 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fd4bffef80542618e90e5cec6d7891d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:57 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:54574
I20260620 20:13:57.343955 32071 tablet_replica.cc:330] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835: stopping tablet replica
I20260620 20:13:57.344254 32071 raft_consensus.cc:2229] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:13:57.344640 32071 raft_consensus.cc:2258] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:57.345916 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 30739
I20260620 20:13:57.346848 32071 ts_tablet_manager.cc:1905] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:57.348814 32070 tablet_replica.cc:330] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add: stopping tablet replica
I20260620 20:13:57.349139 32070 raft_consensus.cc:2229] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:57.349447 32070 raft_consensus.cc:2258] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:57.349977 32071 ts_tablet_manager.cc:1918] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:57.350116 32071 log.cc:1196] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-0/wal/wals/2fd4bffef80542618e90e5cec6d7891d
I20260620 20:13:57.350510 32071 ts_tablet_manager.cc:1939] T 2fd4bffef80542618e90e5cec6d7891d P d1a7fa0094974f778502f3d8de7c5835: Deleting consensus metadata
I20260620 20:13:57.351430 32070 ts_tablet_manager.cc:1905] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:57.351905 32069 tablet_replica.cc:330] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107: stopping tablet replica
I20260620 20:13:57.352223 32069 raft_consensus.cc:2229] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 20:13:57.352473 31621 connection.cc:535] server connection from 127.0.0.1:54574 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:13:57.353555 32069 raft_consensus.cc:2258] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:57.355844 32070 ts_tablet_manager.cc:1918] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:57.355943 32070 log.cc:1196] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/2fd4bffef80542618e90e5cec6d7891d
I20260620 20:13:57.356257 32070 ts_tablet_manager.cc:1939] T 2fd4bffef80542618e90e5cec6d7891d P b6deecfe720547eca4dd1bd60cd96add: Deleting consensus metadata
W20260620 20:13:57.357826 31896 connection.cc:441] server connection from 127.0.0.1:37866 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37866 (request call id 1) could send its response
I20260620 20:13:57.358726 32069 ts_tablet_manager.cc:1905] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:57.361465 32069 ts_tablet_manager.cc:1918] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:57.361552 32069 log.cc:1196] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/2fd4bffef80542618e90e5cec6d7891d
I20260620 20:13:57.361660 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.344865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.24.20.189:39307)
0620 20:13:57.344966 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.344973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.349915 (+ 4942us) server_negotiation.cc:407] Connection header received
0620 20:13:57.351780 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.351783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.351839 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.351923 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:57.353747 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.354605 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.355904 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.356299 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.356425 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.358196 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:57.358200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:57.358202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:57.358228 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:57.359869 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.360342 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.360346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.360348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.360433 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.360768 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.360771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.360773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.360979 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:57.361114 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.361353 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.361466 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 20:13:57.362118 32069 ts_tablet_manager.cc:1939] T 2fd4bffef80542618e90e5cec6d7891d P bb509074a785475ea0930e20cc2a2107: Deleting consensus metadata
W20260620 20:13:57.363142 31761 connection.cc:441] server connection from 127.0.0.1:60076 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60076 (request call id 1) could send its response
I20260620 20:13:57.372745 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.344858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.24.20.188:45015)
0620 20:13:57.344960 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.344966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.362052 (+ 17086us) server_negotiation.cc:407] Connection header received
0620 20:13:57.362266 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.362269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.362322 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.362374 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:57.363234 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.364039 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.365335 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.365700 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.365783 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.368006 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:57.368011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:57.368014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:57.368044 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:57.370758 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.371233 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.371237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.371239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.371318 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.371702 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.371705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.371707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.371904 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:57.372002 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.372335 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.372493 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":28}
I20260620 20:13:57.373394 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.356976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.24.20.189:39307)
0620 20:13:57.357104 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.357108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.363681 (+ 6573us) server_negotiation.cc:407] Connection header received
0620 20:13:57.363848 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.363851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.363894 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.363943 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:57.364898 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.365701 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.367084 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.367528 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.367619 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.369589 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:57.369592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:57.369595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:57.369617 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:57.371381 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.371940 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.371947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.371949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.372034 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.372346 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.372352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.372355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.372620 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:57.372732 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.373112 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.373232 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:13:57.374883 32021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:57.363757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.24.20.188:45015)
0620 20:13:57.363867 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:57.363872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:57.365086 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 20:13:57.365240 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:57.365243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:57.365283 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:57.365329 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:13:57.366169 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.366930 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.368116 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:57.368451 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:57.368533 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:57.370445 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:13:57.370449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:13:57.370451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:13:57.370487 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:13:57.372706 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.373294 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.373298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.373300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.373375 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:13:57.373735 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:13:57.373740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:13:57.373745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:13:57.373939 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:13:57.374084 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:57.374648 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 20:13:57.374757 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":513,"server-negotiator.queue_time_us":48}
I20260620 20:13:58.334864 32080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.334377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.129:48613)
0620 20:13:58.334658 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.334742 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
W20260620 20:13:58.335212 31735 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:44703 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
I20260620 20:13:58.336570 32081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.336067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.131:60987)
0620 20:13:58.336383 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.336471 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":64,"threads_started":1}
W20260620 20:13:58.337101 32008 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:44703 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
I20260620 20:13:58.338948 32082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.338576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.130:47623)
0620 20:13:58.338773 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.338855 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
W20260620 20:13:58.339265 31872 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:44703 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
I20260620 20:13:58.843252 32083 raft_consensus.cc:482] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0bf847e90b54356bcdd594d09491430)
I20260620 20:13:58.843480 32083 raft_consensus.cc:504] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } }
I20260620 20:13:58.844456 32084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0bf847e90b54356bcdd594d09491430)
I20260620 20:13:58.844527 32083 leader_election.cc:288] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0bf847e90b54356bcdd594d09491430 (127.24.20.190:44703), 08260900a88b4eb59a21b4a0071e85f3 (127.24.20.188:45015)
I20260620 20:13:58.844707 32084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } }
I20260620 20:13:58.845207 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.844646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:37762)
0620 20:13:58.845005 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.845104 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":63,"threads_started":1}
W20260620 20:13:58.845459 31355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
I20260620 20:13:58.845727 32084 leader_election.cc:288] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0bf847e90b54356bcdd594d09491430 (127.24.20.190:44703), bab681a79f3d420381d8a9edc451a543 (127.24.20.189:39307)
I20260620 20:13:58.846136 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.845777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:37776)
0620 20:13:58.845975 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.846044 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
W20260620 20:13:58.846304 31539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
I20260620 20:13:58.846621 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.846325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:37792)
0620 20:13:58.846500 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.846551 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
W20260620 20:13:58.846740 31355 leader_election.cc:334] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0bf847e90b54356bcdd594d09491430 (127.24.20.190:44703): Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
I20260620 20:13:58.847368 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.847105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:37798)
0620 20:13:58.847222 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.847279 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 20:13:58.848166 31539 leader_election.cc:334] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0bf847e90b54356bcdd594d09491430 (127.24.20.190:44703): Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
I20260620 20:13:58.849432 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.844995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:45015 (local address 127.0.0.1:43042)
0620 20:13:58.845239 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.845255 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:58.845361 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:58.845657 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:58.845660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:58.845680 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:58.845906 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.845915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.847359 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.847363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.848319 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.848329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.849058 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.849062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.849154 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.849180 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 20:13:58.849268 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 20:13:58.849345 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":51,"threads_started":1}
I20260620 20:13:58.849418 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.845163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.24.20.188:45015)
0620 20:13:58.845398 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:58.845404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:58.845436 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:13:58.845490 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:58.845492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:58.845529 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:58.845627 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:58.846056 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.847255 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.848430 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.848953 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.849037 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.849082 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:58.849281 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 20:13:58.849336 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
I20260620 20:13:58.849897 31565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bab681a79f3d420381d8a9edc451a543" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08260900a88b4eb59a21b4a0071e85f3" is_pre_election: true
I20260620 20:13:58.850093 31565 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bab681a79f3d420381d8a9edc451a543 in term 1.
I20260620 20:13:58.850433 31355 leader_election.cc:302] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08260900a88b4eb59a21b4a0071e85f3, bab681a79f3d420381d8a9edc451a543; no voters: c0bf847e90b54356bcdd594d09491430
I20260620 20:13:58.850674 32083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 20:13:58.850728 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.845948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39307 (local address 127.0.0.1:35030)
0620 20:13:58.846682 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.846695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:58.846784 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:58.847109 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:58.847112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:58.847134 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:58.847269 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.847274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.848139 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.848142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.849183 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.849192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.850324 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.850327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.850426 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.850453 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 20:13:58.850537 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 20:13:58.850605 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":46,"thread_start_us":59,"threads_started":1}
I20260620 20:13:58.850741 32083 raft_consensus.cc:482] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0bf847e90b54356bcdd594d09491430)
I20260620 20:13:58.850781 32083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:13:58.851821 32083 raft_consensus.cc:504] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } }
I20260620 20:13:58.852102 32083 leader_election.cc:288] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [CANDIDATE]: Term 2 election: Requested vote from peers c0bf847e90b54356bcdd594d09491430 (127.24.20.190:44703), 08260900a88b4eb59a21b4a0071e85f3 (127.24.20.188:45015)
I20260620 20:13:58.850775 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.846235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.24.20.189:39307)
0620 20:13:58.846899 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:58.846908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:58.846921 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:13:58.846959 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:58.846962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:58.846999 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:58.847075 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:58.847396 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.848024 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.849615 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.850155 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.850262 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.850340 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:58.850563 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 20:13:58.850626 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":605,"thread_start_us":93,"threads_started":1}
I20260620 20:13:58.852531 31381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08260900a88b4eb59a21b4a0071e85f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bab681a79f3d420381d8a9edc451a543" is_pre_election: true
I20260620 20:13:58.852624 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.852337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:37804)
0620 20:13:58.852490 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.852544 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 20:13:58.852715 31565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bab681a79f3d420381d8a9edc451a543" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08260900a88b4eb59a21b4a0071e85f3"
I20260620 20:13:58.852789 31381 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 08260900a88b4eb59a21b4a0071e85f3 in current term 2: Already voted for candidate bab681a79f3d420381d8a9edc451a543 in this term.
I20260620 20:13:58.852834 31565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:13:58.853192 31541 leader_election.cc:302] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08260900a88b4eb59a21b4a0071e85f3; no voters: bab681a79f3d420381d8a9edc451a543, c0bf847e90b54356bcdd594d09491430
I20260620 20:13:58.853725 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.853547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:37812)
0620 20:13:58.853627 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.853682 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 20:13:58.853817 31355 leader_election.cc:334] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c0bf847e90b54356bcdd594d09491430 (127.24.20.190:44703): Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
I20260620 20:13:58.853996 31565 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bab681a79f3d420381d8a9edc451a543 in term 2.
I20260620 20:13:58.854164 32084 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:13:58.854287 31355 leader_election.cc:302] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08260900a88b4eb59a21b4a0071e85f3, bab681a79f3d420381d8a9edc451a543; no voters: c0bf847e90b54356bcdd594d09491430
I20260620 20:13:58.854425 32083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:13:58.854663 32083 raft_consensus.cc:686] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [term 2 LEADER]: Becoming Leader. State: Replica: bab681a79f3d420381d8a9edc451a543, State: Running, Role: LEADER
I20260620 20:13:58.854898 32083 consensus_queue.cc:237] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } }
I20260620 20:13:58.856258 32094 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bab681a79f3d420381d8a9edc451a543. Latest consensus state: current_term: 2 leader_uuid: "bab681a79f3d420381d8a9edc451a543" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } } }
I20260620 20:13:58.856365 32094 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:58.856727 32096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:13:58.858767 32096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:13:58.859155 32096 catalog_manager.cc:1259] Loaded cluster ID: 4ba2f43188c24306b8eda927b1c49d9b
I20260620 20:13:58.859200 32096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:13:58.859763 32096 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:13:58.860144 32096 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: Loaded TSK: 0
I20260620 20:13:58.860869 32096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:13:58.883839 31370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:13:58.887020 31370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:13:58.890015 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.889784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:37828)
0620 20:13:58.889903 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.889953 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 20:13:58.890033 31565 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [term 2 FOLLOWER]: Refusing update from remote peer bab681a79f3d420381d8a9edc451a543: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 20:13:58.890642 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.890431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.0.0.1:37836)
0620 20:13:58.890520 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.890591 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 20:13:58.890683 32094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 20:13:58.890841 31355 consensus_peers.cc:487] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 -> Peer c0bf847e90b54356bcdd594d09491430 (127.24.20.190:44703): Couldn't send request to peer c0bf847e90b54356bcdd594d09491430. Status: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 20:13:58.892051 32084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bab681a79f3d420381d8a9edc451a543. Latest consensus state: current_term: 2 leader_uuid: "bab681a79f3d420381d8a9edc451a543" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } } }
I20260620 20:13:58.892184 32084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:13:58.893262 32094 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "bab681a79f3d420381d8a9edc451a543" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } } }
I20260620 20:13:58.893378 32094 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543 [sys.catalog]: This master's current role is: LEADER
I20260620 20:13:58.893712 32084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "bab681a79f3d420381d8a9edc451a543" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c0bf847e90b54356bcdd594d09491430" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44703 } } peers { permanent_uuid: "bab681a79f3d420381d8a9edc451a543" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39307 } attrs { promote: false } } peers { permanent_uuid: "08260900a88b4eb59a21b4a0071e85f3" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 45015 } attrs { promote: false } } }
I20260620 20:13:58.893836 32084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08260900a88b4eb59a21b4a0071e85f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:13:58.894737 32098 mvcc.cc:205] Tried to move back new op lower bound from 7299016453688676352 to 7299016453552091136. Current Snapshot: MvccSnapshot[applied={T|T < 7299016453688676352}]
I20260620 20:13:58.903236 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.898943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:44259 (local address 127.0.0.1:37874)
0620 20:13:58.899082 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.899096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:58.899169 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:58.899664 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:58.899667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:58.899684 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:58.899799 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.899809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.900686 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.900689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.901959 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.901969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.902859 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.902862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.902960 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.902987 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 20:13:58.903033 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 20:13:58.903099 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":60}
I20260620 20:13:58.903802 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.899560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:41279 (local address 127.0.0.1:54588)
0620 20:13:58.899691 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.899704 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:58.899772 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:58.900317 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:58.900320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:58.900337 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:58.900415 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.900421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.901480 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.901483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.902555 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.902563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.903423 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.903427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.903513 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.903539 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 20:13:58.903623 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 20:13:58.903687 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":154}
I20260620 20:13:58.903838 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.899861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.24.20.129:41279)
0620 20:13:58.900070 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:58.900075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:58.900092 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:13:58.900148 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:58.900151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:58.900205 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:58.900292 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:58.900547 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.901364 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.902668 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.903291 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.903422 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.903482 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:58.903638 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 20:13:58.903717 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":65,"threads_started":1}
I20260620 20:13:58.904066 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.899577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:44783 (local address 127.0.0.1:60092)
0620 20:13:58.900057 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.900071 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:58.900155 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:58.900740 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:58.900743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:58.900758 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:58.900843 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.900849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.901592 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.901596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.902839 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.902847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.903714 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.903718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.903801 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.903822 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:13:58.903871 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 20:13:58.903957 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":42}
I20260620 20:13:58.904218 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.899151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.24.20.131:44259)
0620 20:13:58.899407 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:58.899412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:58.899427 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:13:58.899487 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:58.899493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:58.899562 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:58.899649 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:58.899900 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.900542 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.902086 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.902727 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.903676 (+ 949us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.903741 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:58.903774 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 20:13:58.903843 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
I20260620 20:13:58.904414 31668 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4653bec5684674bcf229b886f4c645 (DEFAULT_TABLE table=test-table [id=d0ac3d3de7e848508666e137f0782ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:58.904538 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.899842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.24.20.130:44783)
0620 20:13:58.900502 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:58.900507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:58.900531 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:13:58.900578 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:58.900581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:58.900652 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:58.900732 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:58.900955 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.901474 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.902960 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.903573 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.904240 (+ 667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.904309 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:58.904347 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 20:13:58.904414 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":66,"threads_started":1}
I20260620 20:13:58.904742 31668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4653bec5684674bcf229b886f4c645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:58.904687 31941 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4653bec5684674bcf229b886f4c645 (DEFAULT_TABLE table=test-table [id=d0ac3d3de7e848508666e137f0782ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:58.904915 31941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4653bec5684674bcf229b886f4c645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:58.907310 32106 tablet_bootstrap.cc:492] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835: Bootstrap starting.
I20260620 20:13:58.907631 31805 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4653bec5684674bcf229b886f4c645 (DEFAULT_TABLE table=test-table [id=d0ac3d3de7e848508666e137f0782ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:13:58.907831 31805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4653bec5684674bcf229b886f4c645. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:13:58.908166 32106 tablet_bootstrap.cc:654] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:58.908722 32107 tablet_bootstrap.cc:492] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add: Bootstrap starting.
I20260620 20:13:58.909307 32106 tablet_bootstrap.cc:492] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835: No bootstrap required, opened a new log
I20260620 20:13:58.909379 32106 ts_tablet_manager.cc:1397] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:13:58.909626 32107 tablet_bootstrap.cc:654] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:58.909798 32109 tablet_bootstrap.cc:492] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107: Bootstrap starting.
I20260620 20:13:58.909852 32106 raft_consensus.cc:348] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } } peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } }
I20260620 20:13:58.909963 32106 raft_consensus.cc:374] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:58.910014 32106 raft_consensus.cc:729] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a7fa0094974f778502f3d8de7c5835, State: Initialized, Role: FOLLOWER
I20260620 20:13:58.910132 32106 consensus_queue.cc:260] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } } peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } }
I20260620 20:13:58.910475 32106 ts_tablet_manager.cc:1428] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:58.911001 32109 tablet_bootstrap.cc:654] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107: Neither blocks nor log segments found. Creating new log.
I20260620 20:13:58.912210 32109 tablet_bootstrap.cc:492] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107: No bootstrap required, opened a new log
I20260620 20:13:58.912307 32109 ts_tablet_manager.cc:1397] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 20:13:58.912693 32109 raft_consensus.cc:348] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } } peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } }
I20260620 20:13:58.912807 32109 raft_consensus.cc:374] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:58.912842 32109 raft_consensus.cc:729] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb509074a785475ea0930e20cc2a2107, State: Initialized, Role: FOLLOWER
I20260620 20:13:58.912945 31736 heartbeater.cc:503] Master 127.24.20.189:39307 was elected leader, sending a full tablet report...
I20260620 20:13:58.912969 32109 consensus_queue.cc:260] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } } peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } }
I20260620 20:13:58.913923 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.913568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.130:46169)
0620 20:13:58.913772 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.913831 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260620 20:13:58.914362 32109 ts_tablet_manager.cc:1428] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:13:58.914897 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.914679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.130:58931)
0620 20:13:58.914788 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.914838 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 20:13:58.914909 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.911123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.129:37321)
0620 20:13:58.911621 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.911676 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":81,"threads_started":1}
I20260620 20:13:58.914959 31873 heartbeater.cc:503] Master 127.24.20.189:39307 was elected leader, sending a full tablet report...
I20260620 20:13:58.916162 32107 tablet_bootstrap.cc:492] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add: No bootstrap required, opened a new log
I20260620 20:13:58.916252 32107 ts_tablet_manager.cc:1397] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260620 20:13:58.916644 32107 raft_consensus.cc:348] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } } peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } }
I20260620 20:13:58.916764 32107 raft_consensus.cc:374] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:13:58.916800 32107 raft_consensus.cc:729] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6deecfe720547eca4dd1bd60cd96add, State: Initialized, Role: FOLLOWER
I20260620 20:13:58.916925 32107 consensus_queue.cc:260] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } } peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } }
I20260620 20:13:58.917265 32107 ts_tablet_manager.cc:1428] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:13:58.917986 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.917531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.131:47553)
0620 20:13:58.917808 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.917864 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":85,"threads_started":1}
I20260620 20:13:58.918507 32009 heartbeater.cc:503] Master 127.24.20.189:39307 was elected leader, sending a full tablet report...
I20260620 20:13:58.918592 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.918393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.131:41001)
0620 20:13:58.918476 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.918532 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 20:13:58.951900 32116 raft_consensus.cc:482] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:13:58.952028 32116 raft_consensus.cc:504] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } } peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } }
I20260620 20:13:58.953020 32116 leader_election.cc:288] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb509074a785475ea0930e20cc2a2107 (127.24.20.130:44783), d1a7fa0094974f778502f3d8de7c5835 (127.24.20.129:41279)
I20260620 20:13:58.957122 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.953444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:55045 (local address 127.24.20.130:44783)
0620 20:13:58.953572 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:58.953576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:58.953587 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:13:58.953653 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:58.953656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:58.953704 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:58.953786 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:58.954133 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.954762 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.956025 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.956468 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.956561 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.956655 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:58.956903 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 20:13:58.956996 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:13:58.957118 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.953372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:51309 (local address 127.24.20.129:41279)
0620 20:13:58.953493 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:58.953498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:58.953846 (+ 348us) server_negotiation.cc:407] Connection header received
0620 20:13:58.953973 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:58.953979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:58.954031 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:58.954110 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:13:58.954337 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.955041 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.956165 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.956572 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.956702 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.956750 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:58.956905 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 20:13:58.956986 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:13:58.957209 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.953218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:41279 (local address 127.24.20.131:51309)
0620 20:13:58.953751 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.953766 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:58.953873 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:58.954129 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:58.954132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:58.954148 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:58.954242 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.954248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.955159 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.955162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.956034 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.956043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.956699 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.956703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.956792 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.956808 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 20:13:58.956890 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 20:13:58.957036 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":302,"spinlock_wait_cycles":2944,"thread_start_us":110,"threads_started":1}
I20260620 20:13:58.957350 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.953218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:44783 (local address 127.24.20.131:55045)
0620 20:13:58.953306 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.953322 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:58.953416 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:58.953809 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:13:58.953828 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:13:58.953872 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:13:58.954013 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.954021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.954845 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.954848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.955911 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:13:58.955919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.956584 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.956612 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:13:58.956740 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.956795 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 20:13:58.956888 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 20:13:58.957009 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":43,"spinlock_wait_cycles":12416}
I20260620 20:13:58.957762 31825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4653bec5684674bcf229b886f4c645" candidate_uuid: "b6deecfe720547eca4dd1bd60cd96add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb509074a785475ea0930e20cc2a2107" is_pre_election: true
I20260620 20:13:58.957726 31688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4653bec5684674bcf229b886f4c645" candidate_uuid: "b6deecfe720547eca4dd1bd60cd96add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a7fa0094974f778502f3d8de7c5835" is_pre_election: true
I20260620 20:13:58.957890 31825 raft_consensus.cc:2454] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6deecfe720547eca4dd1bd60cd96add in term 0.
I20260620 20:13:58.957952 31688 raft_consensus.cc:2454] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6deecfe720547eca4dd1bd60cd96add in term 0.
I20260620 20:13:58.958284 31895 leader_election.cc:302] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b6deecfe720547eca4dd1bd60cd96add, bb509074a785475ea0930e20cc2a2107, d1a7fa0094974f778502f3d8de7c5835; no voters:
I20260620 20:13:58.958572 32116 raft_consensus.cc:2781] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:13:58.958644 32116 raft_consensus.cc:482] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:13:58.958698 32116 raft_consensus.cc:3037] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:58.959659 32116 raft_consensus.cc:504] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } } peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } }
I20260620 20:13:58.960011 32116 leader_election.cc:288] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [CANDIDATE]: Term 1 election: Requested vote from peers bb509074a785475ea0930e20cc2a2107 (127.24.20.130:44783), d1a7fa0094974f778502f3d8de7c5835 (127.24.20.129:41279)
I20260620 20:13:58.960314 31825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4653bec5684674bcf229b886f4c645" candidate_uuid: "b6deecfe720547eca4dd1bd60cd96add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb509074a785475ea0930e20cc2a2107"
I20260620 20:13:58.960345 31688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4653bec5684674bcf229b886f4c645" candidate_uuid: "b6deecfe720547eca4dd1bd60cd96add" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1a7fa0094974f778502f3d8de7c5835"
I20260620 20:13:58.960436 31825 raft_consensus.cc:3037] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:58.960461 31688 raft_consensus.cc:3037] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:13:58.961308 31825 raft_consensus.cc:2454] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6deecfe720547eca4dd1bd60cd96add in term 1.
I20260620 20:13:58.961490 31688 raft_consensus.cc:2454] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6deecfe720547eca4dd1bd60cd96add in term 1.
I20260620 20:13:58.961567 31895 leader_election.cc:302] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6deecfe720547eca4dd1bd60cd96add, bb509074a785475ea0930e20cc2a2107; no voters:
I20260620 20:13:58.961854 32116 raft_consensus.cc:2781] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:13:58.962024 32116 raft_consensus.cc:686] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [term 1 LEADER]: Becoming Leader. State: Replica: b6deecfe720547eca4dd1bd60cd96add, State: Running, Role: LEADER
I20260620 20:13:58.962486 32116 consensus_queue.cc:237] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } } peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } }
I20260620 20:13:58.962649 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.962426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.131:32939)
0620 20:13:58.962528 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.962575 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 20:13:58.964880 31371 catalog_manager.cc:5495] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add reported cstate change: term changed from 0 to 1, leader changed from <none> to b6deecfe720547eca4dd1bd60cd96add (127.24.20.131). New cstate: current_term: 1 leader_uuid: "b6deecfe720547eca4dd1bd60cd96add" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6deecfe720547eca4dd1bd60cd96add" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 44259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 } health_report { overall_health: UNKNOWN } } }
I20260620 20:13:58.980576 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.977595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.24.20.131:44259)
0620 20:13:58.977714 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 20:13:58.977719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:13:58.977736 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:13:58.977814 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:13:58.977817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:13:58.977862 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:13:58.977947 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:13:58.978125 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.978617 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.979365 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:13:58.979582 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:13:58.979659 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:13:58.979845 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:13:58.980232 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:13:58.980314 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 20:13:58.980396 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 20:13:58.980454 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:13:58.982883 31825 raft_consensus.cc:1261] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107 [term 1 FOLLOWER]: Refusing update from remote peer b6deecfe720547eca4dd1bd60cd96add: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:58.983146 31688 raft_consensus.cc:1261] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835 [term 1 FOLLOWER]: Refusing update from remote peer b6deecfe720547eca4dd1bd60cd96add: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:13:58.983321 32119 consensus_queue.cc:1035] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb509074a785475ea0930e20cc2a2107" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:58.983608 32116 consensus_queue.cc:1035] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a7fa0094974f778502f3d8de7c5835" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:13:58.985404 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.984906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.130:44545)
0620 20:13:58.985282 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.985336 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
I20260620 20:13:58.985404 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.985153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.129:48395)
0620 20:13:58.985291 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.985336 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 20:13:58.988304 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.987932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.131:42145)
0620 20:13:58.988188 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.988240 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260620 20:13:58.988933 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.988715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.129:49129)
0620 20:13:58.988821 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.988870 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 20:13:58.989547 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.989219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.131:60357)
0620 20:13:58.989438 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.989484 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260620 20:13:58.989884 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:13:58.989437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44703 (local address 127.24.20.130:40221)
0620 20:13:58.989773 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:58.989822 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276}
I20260620 20:13:58.996858 31370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:58.997020 31370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:13:59.000583 31370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bab681a79f3d420381d8a9edc451a543: Sending DeleteTablet for 3 replicas of tablet 5f4653bec5684674bcf229b886f4c645
I20260620 20:13:59.001299 31941 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f4653bec5684674bcf229b886f4c645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:58 UTC) from {username='oryx', principal='oryx/127.24.20.189@KRBTEST.COM'} at 127.0.0.1:37874
I20260620 20:13:59.001421 31668 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f4653bec5684674bcf229b886f4c645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:58 UTC) from {username='oryx', principal='oryx/127.24.20.189@KRBTEST.COM'} at 127.0.0.1:54588
I20260620 20:13:59.001726 32133 tablet_replica.cc:330] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835: stopping tablet replica
I20260620 20:13:59.001857 32133 raft_consensus.cc:2229] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:59.001840 31805 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f4653bec5684674bcf229b886f4c645 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:13:58 UTC) from {username='oryx', principal='oryx/127.24.20.189@KRBTEST.COM'} at 127.0.0.1:60092
I20260620 20:13:59.001986 32133 raft_consensus.cc:2258] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:59.002492 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 31516
I20260620 20:13:59.002751 32133 ts_tablet_manager.cc:1905] T 5f4653bec5684674bcf229b886f4c645 P d1a7fa0094974f778502f3d8de7c5835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:59.003037 32134 tablet_replica.cc:330] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107: stopping tablet replica
I20260620 20:13:59.003159 32134 raft_consensus.cc:2229] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:13:59.003295 32134 raft_consensus.cc:2258] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:59.003846 32134 ts_tablet_manager.cc:1905] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:59.006573 32134 ts_tablet_manager.cc:1918] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:59.006660 32134 log.cc:1196] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/5f4653bec5684674bcf229b886f4c645
I20260620 20:13:59.006938 32134 ts_tablet_manager.cc:1939] T 5f4653bec5684674bcf229b886f4c645 P bb509074a785475ea0930e20cc2a2107: Deleting consensus metadata
I20260620 20:13:59.007623 31356 catalog_manager.cc:4841] TS bb509074a785475ea0930e20cc2a2107 (127.24.20.130:44783): tablet 5f4653bec5684674bcf229b886f4c645 (table test-table [id=d0ac3d3de7e848508666e137f0782ea4]) successfully deleted
I20260620 20:13:59.011396 32132 tablet_replica.cc:330] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add: stopping tablet replica
W20260620 20:13:59.011503 32092 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:13:59.011017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:41279 (local address 127.0.0.1:54604)
0620 20:13:59.011141 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 20:13:59.011158 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:13:59.011246 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:13:59.011379 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:41279: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 20:13:59.011637 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 31742
I20260620 20:13:59.011669 32132 raft_consensus.cc:2229] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [term 1 LEADER]: Raft consensus shutting down.
W20260620 20:13:59.011828 31357 catalog_manager.cc:4584] TS d1a7fa0094974f778502f3d8de7c5835 (127.24.20.129:41279): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f4653bec5684674bcf229b886f4c645: Network error: Client connection negotiation failed: client connection to 127.24.20.129:41279: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260620 20:13:59.012216 32132 raft_consensus.cc:2258] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:13:59.013456 32132 ts_tablet_manager.cc:1905] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:13:59.015407 32132 ts_tablet_manager.cc:1918] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:13:59.015499 32132 log.cc:1196] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/5f4653bec5684674bcf229b886f4c645
I20260620 20:13:59.015839 32132 ts_tablet_manager.cc:1939] T 5f4653bec5684674bcf229b886f4c645 P b6deecfe720547eca4dd1bd60cd96add: Deleting consensus metadata
I20260620 20:13:59.016436 31355 catalog_manager.cc:4841] TS b6deecfe720547eca4dd1bd60cd96add (127.24.20.131:44259): tablet 5f4653bec5684674bcf229b886f4c645 (table test-table [id=d0ac3d3de7e848508666e137f0782ea4]) successfully deleted
I20260620 20:13:59.019433 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 31878
I20260620 20:13:59.024845 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 31235
I20260620 20:13:59.031078 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 31343
2026-06-20T20:13:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5194 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:13:59 dist-test-slave-pflr krb5kdc[32137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:13:59 dist-test-slave-pflr krb5kdc[32137](info): set up 2 sockets
Jun 20 20:13:59 dist-test-slave-pflr krb5kdc[32137](info): commencing operation
krb5kdc: starting...
W20260620 20:14:00.077642 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:14:00 dist-test-slave-pflr krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.190.keytab.
Entry for principal oryx/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.190.keytab.
I20260620 20:14:00.194931 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:38973
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.190.keytab
--principal=oryx/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:38973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:00.302165 32150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:00.302434 32150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:00.302508 32150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:00.306521 32150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:00.306571 32150 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:14:00.306588 32150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:00.306607 32150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:00.306661 32150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:00.306689 32150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:00.311902 32150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:38973
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.190.keytab
--principal=oryx/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:00.312901 32150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:00.313905 32150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:00.322311 32156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:00.322324 32155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:00.322324 32158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:00.322398 32150 server_base.cc:1034] running on GCE node
I20260620 20:14:00.323144 32150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:00.324210 32150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:00.325388 32150 hybrid_clock.cc:648] HybridClock initialized: now 1781986440325383 us; error 33 us; skew 500 ppm
Jun 20 20:14:00 dist-test-slave-pflr krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986440, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:00.331197 32150 init.cc:377] Logged in from keytab as oryx/127.24.20.190@KRBTEST.COM (short username oryx)
I20260620 20:14:00.332042 32150 webserver.cc:460] Webserver started at http://127.24.20.190:36619/ using document root <none> and password file <none>
I20260620 20:14:00.332578 32150 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:00.332710 32150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:00.332922 32150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:00.334913 32150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "e742953c59a74113ae92276324d9b995"
format_stamp: "Formatted at 2026-06-20 20:14:00 on dist-test-slave-pflr"
server_key: "7e7d39c1b1a674dc57de9bdcbcbb9a6e"
server_key_iv: "e7d9400b00664f31172183a83b57f915"
server_key_version: "encryptionkey@0"
I20260620 20:14:00.335455 32150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "e742953c59a74113ae92276324d9b995"
format_stamp: "Formatted at 2026-06-20 20:14:00 on dist-test-slave-pflr"
server_key: "7e7d39c1b1a674dc57de9bdcbcbb9a6e"
server_key_iv: "e7d9400b00664f31172183a83b57f915"
server_key_version: "encryptionkey@0"
I20260620 20:14:00.339143 32150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 20:14:00.341456 32165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:00.342599 32150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:14:00.342731 32150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "e742953c59a74113ae92276324d9b995"
format_stamp: "Formatted at 2026-06-20 20:14:00 on dist-test-slave-pflr"
server_key: "7e7d39c1b1a674dc57de9bdcbcbb9a6e"
server_key_iv: "e7d9400b00664f31172183a83b57f915"
server_key_version: "encryptionkey@0"
I20260620 20:14:00.342831 32150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:00.357336 32150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:00.360016 32150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:00.360199 32150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:00.367624 32150 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:38973
I20260620 20:14:00.367645 32227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:38973 every 8 connection(s)
I20260620 20:14:00.368477 32150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:00.370442 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 32150
I20260620 20:14:00.370601 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:14:00.370931 24658 external_mini_cluster.cc:1428] Setting key 545713eb9b8c5ef67df4b1f69691b044
I20260620 20:14:00.371551 32228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:00.377328 32228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995: Bootstrap starting.
Jun 20 20:14:00 dist-test-slave-pflr krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
I20260620 20:14:00.379870 32228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:00.381302 32228 log.cc:824] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:00.383494 32228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995: No bootstrap required, opened a new log
I20260620 20:14:00.384701 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.372284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.24.20.190:38973)
0620 20:14:00.372735 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:00.372749 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:00.372782 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:14:00.373390 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:00.373405 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:00.373702 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:00.374008 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:00.374792 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.375737 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.376556 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.377048 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.377211 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:00.379927 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:00.379947 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:00.379965 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:00.380002 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:00.381924 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:00.382497 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:00.382501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:00.382505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:00.382631 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:00.382920 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:00.382922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:00.382923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:00.383251 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:00.383376 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:00.383734 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 20:14:00.384005 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":128,"threads_started":1}
I20260620 20:14:00.386703 32228 raft_consensus.cc:348] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e742953c59a74113ae92276324d9b995" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 38973 } }
I20260620 20:14:00.386909 32228 raft_consensus.cc:374] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:00.386989 32228 raft_consensus.cc:729] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e742953c59a74113ae92276324d9b995, State: Initialized, Role: FOLLOWER
I20260620 20:14:00.387390 32228 consensus_queue.cc:260] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e742953c59a74113ae92276324d9b995" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 38973 } }
I20260620 20:14:00.387543 32228 raft_consensus.cc:388] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:00.387630 32228 raft_consensus.cc:482] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:00.387746 32228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:00.388670 32228 raft_consensus.cc:504] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e742953c59a74113ae92276324d9b995" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 38973 } }
I20260620 20:14:00.389000 32228 leader_election.cc:302] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e742953c59a74113ae92276324d9b995; no voters:
I20260620 20:14:00.389324 32228 leader_election.cc:288] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:00.389631 32233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:00.389820 32233 raft_consensus.cc:686] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [term 1 LEADER]: Becoming Leader. State: Replica: e742953c59a74113ae92276324d9b995, State: Running, Role: LEADER
I20260620 20:14:00.390127 32233 consensus_queue.cc:237] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e742953c59a74113ae92276324d9b995" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 38973 } }
I20260620 20:14:00.390580 32228 sys_catalog.cc:564] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:00.391752 32235 sys_catalog.cc:455] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e742953c59a74113ae92276324d9b995. Latest consensus state: current_term: 1 leader_uuid: "e742953c59a74113ae92276324d9b995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e742953c59a74113ae92276324d9b995" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 38973 } } }
I20260620 20:14:00.391880 32235 sys_catalog.cc:458] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:00.392211 32234 sys_catalog.cc:455] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e742953c59a74113ae92276324d9b995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e742953c59a74113ae92276324d9b995" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 38973 } } }
I20260620 20:14:00.392364 32234 sys_catalog.cc:458] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:00.392676 32242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:00.395067 32242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:00.399484 32242 catalog_manager.cc:1347] Generated new cluster ID: 78bafb5600094a98aae08971fb185eaa
I20260620 20:14:00.399540 32242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:00.405373 32242 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:00.406148 32242 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:00.413606 32242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995: Generated new TSK 0
I20260620 20:14:00.414168 32242 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab.
Entry for principal oryx/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab.
I20260620 20:14:00.443194 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:38973
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab
--principal=oryx/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:00.565011 32254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:00.565235 32254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:00.565308 32254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:00.569311 32254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:00.569365 32254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:00.569402 32254 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:14:00.569523 32254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:14:00.574592 32254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab
--principal=oryx/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.20.190:38973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:00.575543 32254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:00.576543 32254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:00.585510 32259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:00.585577 32262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:00.585830 32260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:00.585709 32254 server_base.cc:1034] running on GCE node
I20260620 20:14:00.586194 32254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:00.586758 32254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:00.587904 32254 hybrid_clock.cc:648] HybridClock initialized: now 1781986440587876 us; error 43 us; skew 500 ppm
Jun 20 20:14:00 dist-test-slave-pflr krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986440, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:00.590423 32254 init.cc:377] Logged in from keytab as oryx/127.24.20.129@KRBTEST.COM (short username oryx)
I20260620 20:14:00.591255 32254 webserver.cc:460] Webserver started at http://127.24.20.129:43273/ using document root <none> and password file <none>
I20260620 20:14:00.591750 32254 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:00.591806 32254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:00.592065 32254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:00.594029 32254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "fa542aa98fb544329ed5ed189e416d0e"
format_stamp: "Formatted at 2026-06-20 20:14:00 on dist-test-slave-pflr"
server_key: "dd68c9278b2e437c42c4d9b3d49c9de6"
server_key_iv: "464b6f0030410b988ae2fa69353adcd2"
server_key_version: "encryptionkey@0"
I20260620 20:14:00.594502 32254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "fa542aa98fb544329ed5ed189e416d0e"
format_stamp: "Formatted at 2026-06-20 20:14:00 on dist-test-slave-pflr"
server_key: "dd68c9278b2e437c42c4d9b3d49c9de6"
server_key_iv: "464b6f0030410b988ae2fa69353adcd2"
server_key_version: "encryptionkey@0"
I20260620 20:14:00.597894 32254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:14:00.600023 32269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:00.601183 32254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:14:00.601272 32254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "fa542aa98fb544329ed5ed189e416d0e"
format_stamp: "Formatted at 2026-06-20 20:14:00 on dist-test-slave-pflr"
server_key: "dd68c9278b2e437c42c4d9b3d49c9de6"
server_key_iv: "464b6f0030410b988ae2fa69353adcd2"
server_key_version: "encryptionkey@0"
I20260620 20:14:00.601351 32254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:00.622400 32254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:00.625257 32254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:00.625403 32254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:00.626677 32254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:00.626750 32254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:00.626822 32254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:00.626864 32254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:14:00 dist-test-slave-pflr krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986440, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
I20260620 20:14:00.637668 32254 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:44977
I20260620 20:14:00.637686 32384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:44977 every 8 connection(s)
I20260620 20:14:00.638731 32254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:14:00.640928 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.627519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:53041 (local address 127.24.20.190:38973)
0620 20:14:00.627644 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:00.627648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:00.628718 (+ 1070us) server_negotiation.cc:407] Connection header received
0620 20:14:00.629856 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:00.629859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:00.629937 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:00.630002 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:00.631656 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.632579 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.633835 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.634089 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.634203 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:00.636885 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:00.636889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:00.636891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:00.636919 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:00.638633 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:00.639331 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:00.639336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:00.639338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:00.639510 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:00.639919 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:00.639922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:00.639924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:00.640106 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:00.640192 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:00.640687 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 20:14:00.640793 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:14:00.641170 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 32254
I20260620 20:14:00.641309 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:14:00.641619 24658 external_mini_cluster.cc:1428] Setting key f742e30da104695668eef399feb6b7cc
I20260620 20:14:00.641860 32278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.628043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:38973 (local address 127.24.20.129:53041)
0620 20:14:00.628538 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:00.628578 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:00.629583 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:00.630404 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:00.630411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:00.630851 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:00.631476 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:00.631496 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.632720 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.632722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:00.633709 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:00.633718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.634253 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.634257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:00.634414 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:00.634959 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:00.634977 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:00.636722 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:00.638767 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:00.638772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:00.638782 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:00.639193 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:00.639638 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:00.639641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:00.639643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:00.639799 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:00.640198 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:00.640202 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:00.640401 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 20:14:00.640564 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 20:14:00.640855 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":135,"threads_started":1}
I20260620 20:14:00.643214 32385 heartbeater.cc:347] Connected to a master server at 127.24.20.190:38973
I20260620 20:14:00.643353 32385 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:00.643884 32385 heartbeater.cc:511] Master 127.24.20.190:38973 requested a full tablet report, sending...
I20260620 20:14:00.645380 32182 ts_manager.cc:194] Registered new tserver with Master: fa542aa98fb544329ed5ed189e416d0e (127.24.20.129:44977)
I20260620 20:14:00.646977 32182 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:53041
WARNING: no policy specified for oryx/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:14:00.652524 32392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.645376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:38973 (local address 127.24.20.129:53081)
0620 20:14:00.645749 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:00.645763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:00.645877 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:00.646120 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:00.646123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:00.646304 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:00.646391 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:00.646396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.647355 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.647358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:00.648056 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:00.648066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.648524 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.648528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:00.648665 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:00.649155 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:00.649173 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:00.649470 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:00.651108 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:00.651113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:00.651115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:00.651310 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:00.651585 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:00.651590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:00.651592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:00.651675 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:00.652134 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:00.652136 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:00.652205 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 20:14:00.652286 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 20:14:00.652381 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":10,"thread_start_us":164,"threads_started":1}
I20260620 20:14:00.652555 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.645457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:53081 (local address 127.24.20.190:38973)
0620 20:14:00.645592 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:00.645598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:00.645843 (+ 245us) server_negotiation.cc:407] Connection header received
0620 20:14:00.645976 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:00.645979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:00.646024 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:00.646104 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:00.646506 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.647239 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.648167 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.648397 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.648520 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:00.649607 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:00.649611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:00.649613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:00.649636 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:00.651000 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:00.651412 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:00.651415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:00.651417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:00.651500 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:00.651798 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:00.651804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:00.651808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:00.652038 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:00.652135 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:00.652342 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 20:14:00.652428 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.20.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab.
Entry for principal oryx/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab.
I20260620 20:14:00.669401 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:38973
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab
--principal=oryx/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:00.777315 32394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:00.777599 32394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:00.777680 32394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:00.781589 32394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:00.781680 32394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:00.781749 32394 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:14:00.781822 32394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:14:00.786844 32394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab
--principal=oryx/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.20.190:38973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:00.787822 32394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:00.788813 32394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:00.798681 32399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:00.798784 32402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:00.798686 32400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:00.798946 32394 server_base.cc:1034] running on GCE node
I20260620 20:14:00.799485 32394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:00.800115 32394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:00.801301 32394 hybrid_clock.cc:648] HybridClock initialized: now 1781986440801283 us; error 30 us; skew 500 ppm
Jun 20 20:14:00 dist-test-slave-pflr krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986440, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:00.803957 32394 init.cc:377] Logged in from keytab as oryx/127.24.20.130@KRBTEST.COM (short username oryx)
I20260620 20:14:00.804903 32394 webserver.cc:460] Webserver started at http://127.24.20.130:37569/ using document root <none> and password file <none>
I20260620 20:14:00.805459 32394 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:00.805519 32394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:00.805684 32394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:00.807621 32394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74"
format_stamp: "Formatted at 2026-06-20 20:14:00 on dist-test-slave-pflr"
server_key: "7d8549346ebbf1161b0b74941f17fc46"
server_key_iv: "99d19d6f6481d4bbb293c6942ecd7bb6"
server_key_version: "encryptionkey@0"
I20260620 20:14:00.808106 32394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74"
format_stamp: "Formatted at 2026-06-20 20:14:00 on dist-test-slave-pflr"
server_key: "7d8549346ebbf1161b0b74941f17fc46"
server_key_iv: "99d19d6f6481d4bbb293c6942ecd7bb6"
server_key_version: "encryptionkey@0"
I20260620 20:14:00.811630 32394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:14:00.813972 32409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:00.815220 32394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:14:00.815310 32394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74"
format_stamp: "Formatted at 2026-06-20 20:14:00 on dist-test-slave-pflr"
server_key: "7d8549346ebbf1161b0b74941f17fc46"
server_key_iv: "99d19d6f6481d4bbb293c6942ecd7bb6"
server_key_version: "encryptionkey@0"
I20260620 20:14:00.815388 32394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:00.830096 32394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:00.832825 32394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:00.832966 32394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:00.834244 32394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:00.834291 32394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:00.834350 32394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:00.834384 32394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:14:00 dist-test-slave-pflr krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986440, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
I20260620 20:14:00.845402 32394 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:38935
I20260620 20:14:00.845477 32524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:38935 every 8 connection(s)
I20260620 20:14:00.846534 32394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:14:00.848196 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.835073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:54325 (local address 127.24.20.190:38973)
0620 20:14:00.835189 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:00.835194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:00.836230 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 20:14:00.837343 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:00.837347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:00.837408 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:00.837519 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:00.838751 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.839273 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.840983 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.841233 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.841346 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:00.844339 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:00.844343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:00.844346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:00.844377 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:00.846144 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:00.846782 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:00.846785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:00.846787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:00.846884 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:00.847285 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:00.847290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:00.847294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:00.847512 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:00.847587 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:00.847983 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 20:14:00.848078 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 20:14:00.849022 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.835605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:38973 (local address 127.24.20.130:54325)
0620 20:14:00.836091 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:00.836131 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:00.837060 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:00.837687 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:00.837697 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:00.838020 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:00.838589 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:00.838609 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.839393 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.839398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:00.840830 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:00.840839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.841389 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.841393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:00.841605 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:00.842237 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:00.842263 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:00.844159 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:00.846274 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:00.846280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:00.846293 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:00.846682 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:00.847017 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:00.847020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:00.847023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:00.847178 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:00.847594 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:00.847599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:00.847794 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 20:14:00.847939 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 20:14:00.848169 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":135,"threads_started":1}
I20260620 20:14:00.850108 32525 heartbeater.cc:347] Connected to a master server at 127.24.20.190:38973
I20260620 20:14:00.850241 32525 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:00.850822 32525 heartbeater.cc:511] Master 127.24.20.190:38973 requested a full tablet report, sending...
I20260620 20:14:00.851846 32182 ts_manager.cc:194] Registered new tserver with Master: 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130:38935)
I20260620 20:14:00.852715 32182 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:54325
I20260620 20:14:00.855244 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 32394
I20260620 20:14:00.855376 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:14:00.855764 24658 external_mini_cluster.cc:1428] Setting key 57af631e4491db3c31215ebe353dd66c
I20260620 20:14:00.859669 32531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.852311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:38973 (local address 127.24.20.130:41747)
0620 20:14:00.852626 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:00.852642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:00.852764 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:00.853019 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:00.853024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:00.853237 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:00.853351 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:00.853356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.854346 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.854349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:00.854910 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:00.854917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.855492 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.855496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:00.855574 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:00.856059 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:00.856069 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:00.856437 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:00.858170 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:00.858176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:00.858181 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:00.858439 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:00.858747 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:00.858749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:00.858750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:00.858822 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:00.859305 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:00.859307 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:00.859373 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 20:14:00.859448 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 20:14:00.859544 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":114,"threads_started":1}
I20260620 20:14:00.859863 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:00.852426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:41747 (local address 127.24.20.190:38973)
0620 20:14:00.852628 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:00.852634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:00.852718 (+ 84us) server_negotiation.cc:407] Connection header received
0620 20:14:00.852852 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:00.852855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:00.852895 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:00.852990 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:00.853457 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.854203 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.855036 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:00.855357 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:00.855466 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:00.856556 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:00.856560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:00.856562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:00.856585 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:00.858058 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:00.858542 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:00.858547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:00.858549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:00.858646 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:00.858902 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:00.858906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:00.858908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:00.859214 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:00.859324 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:00.859534 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 20:14:00.859655 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
WARNING: no policy specified for oryx/127.24.20.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.20.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab.
Entry for principal oryx/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab.
I20260620 20:14:00.881754 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:38973
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab
--principal=oryx/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:00.989988 32534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:00.990226 32534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:00.990268 32534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:00.994423 32534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:00.994494 32534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:00.994565 32534 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:14:00.994621 32534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:14:00.999696 32534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab
--principal=oryx/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.20.190:38973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:01.000664 32534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:01.001586 32534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:01.010879 32540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:01.010901 32539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:01.010936 32534 server_base.cc:1034] running on GCE node
W20260620 20:14:01.011065 32542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:01.011404 32534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:01.012030 32534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:01.013195 32534 hybrid_clock.cc:648] HybridClock initialized: now 1781986441013164 us; error 45 us; skew 500 ppm
Jun 20 20:14:01 dist-test-slave-pflr krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986441, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:01.015873 32534 init.cc:377] Logged in from keytab as oryx/127.24.20.131@KRBTEST.COM (short username oryx)
I20260620 20:14:01.016758 32534 webserver.cc:460] Webserver started at http://127.24.20.131:33173/ using document root <none> and password file <none>
I20260620 20:14:01.017292 32534 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:01.017373 32534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:01.017588 32534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:01.019690 32534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "f7fadf15df8942e9808a38718ebaa937"
format_stamp: "Formatted at 2026-06-20 20:14:01 on dist-test-slave-pflr"
server_key: "4d1e65cd5a7410107c1063228361af9e"
server_key_iv: "b652b75047b345b543822e99d844e1e2"
server_key_version: "encryptionkey@0"
I20260620 20:14:01.020303 32534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "f7fadf15df8942e9808a38718ebaa937"
format_stamp: "Formatted at 2026-06-20 20:14:01 on dist-test-slave-pflr"
server_key: "4d1e65cd5a7410107c1063228361af9e"
server_key_iv: "b652b75047b345b543822e99d844e1e2"
server_key_version: "encryptionkey@0"
I20260620 20:14:01.023852 32534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 20:14:01.026193 32549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:01.027344 32534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:14:01.027467 32534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "f7fadf15df8942e9808a38718ebaa937"
format_stamp: "Formatted at 2026-06-20 20:14:01 on dist-test-slave-pflr"
server_key: "4d1e65cd5a7410107c1063228361af9e"
server_key_iv: "b652b75047b345b543822e99d844e1e2"
server_key_version: "encryptionkey@0"
I20260620 20:14:01.027563 32534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:01.041251 32534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:01.044139 32534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:01.044312 32534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:01.045745 32534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:01.045792 32534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:01.045848 32534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:01.045878 32534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:01.055287 32534 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:33427
I20260620 20:14:01.055344 32664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:33427 every 8 connection(s)
Jun 20 20:14:01 dist-test-slave-pflr krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986441, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
I20260620 20:14:01.056146 32534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:14:01.057515 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 32534
I20260620 20:14:01.057646 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:14:01.057971 24658 external_mini_cluster.cc:1428] Setting key 67344fe7705e3a3a563a4908a94b85b4
I20260620 20:14:01.059921 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.046439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:45729 (local address 127.24.20.190:38973)
0620 20:14:01.046570 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.046576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.047601 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 20:14:01.048661 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.048667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.048733 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.048830 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:01.050440 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.051311 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.052848 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.053104 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.053237 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.056036 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:01.056040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:01.056043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:01.056069 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:01.057951 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:01.058471 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:01.058475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:01.058477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:01.058567 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:01.058889 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:01.058893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:01.058895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:01.059077 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:01.059174 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:01.059644 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 20:14:01.059758 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:14:01.060668 32558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.047010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:38973 (local address 127.24.20.131:45729)
0620 20:14:01.047466 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:01.047505 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:01.048407 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:01.049138 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:01.049149 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:01.049535 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:01.050218 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.050238 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.051433 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.051438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.052729 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.052739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.053254 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.053258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.053455 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.054111 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:01.054139 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:01.055876 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:01.058098 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:01.058108 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:01.058117 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:01.058352 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:01.058664 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:01.058667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:01.058669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:01.058792 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:01.059182 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:01.059191 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:01.059403 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 20:14:01.059593 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 20:14:01.059881 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":132,"threads_started":1}
I20260620 20:14:01.061707 32665 heartbeater.cc:347] Connected to a master server at 127.24.20.190:38973
I20260620 20:14:01.061834 32665 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:01.062332 32665 heartbeater.cc:511] Master 127.24.20.190:38973 requested a full tablet report, sending...
I20260620 20:14:01.063328 32182 ts_manager.cc:194] Registered new tserver with Master: f7fadf15df8942e9808a38718ebaa937 (127.24.20.131:33427)
I20260620 20:14:01.064075 32182 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:45729
I20260620 20:14:01.070199 32671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.063799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:38973 (local address 127.24.20.131:43121)
0620 20:14:01.064055 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:01.064073 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:01.064171 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:01.064415 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:01.064417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:01.064642 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:01.064750 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.064756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.065690 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.065692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.066239 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.066245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.066625 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.066628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.066694 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.067097 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:01.067107 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:01.067390 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:01.069041 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:01.069043 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:01.069045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:01.069218 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:01.069433 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:01.069435 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:01.069436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:01.069503 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:01.069783 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:01.069785 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:01.069849 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 20:14:01.069924 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 20:14:01.070014 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":94,"threads_started":1}
I20260620 20:14:01.070211 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.063866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:43121 (local address 127.24.20.190:38973)
0620 20:14:01.063983 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.063988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.064144 (+ 156us) server_negotiation.cc:407] Connection header received
0620 20:14:01.064245 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.064248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.064309 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.064381 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:01.064852 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.065580 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.066335 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.066534 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.066611 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.067502 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:01.067505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:01.067507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:01.067526 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:01.068941 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:01.069293 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:01.069296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:01.069297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:01.069362 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:01.069573 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:01.069575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:01.069576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:01.069714 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:01.069784 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:01.069991 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 20:14:01.070096 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:14:01.071375 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:14:01 dist-test-slave-pflr krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:14:01 dist-test-slave-pflr krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:01 dist-test-slave-pflr krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:14:01.095943 32231 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:01.090113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.24.20.190:38973)
0620 20:14:01.090232 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.090235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.090246 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:14:01.090285 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.090287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.090323 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.090409 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:01.091156 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.091706 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.092551 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.092818 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.092925 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.095831 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49300: BlockingRecv error: recv got EOF from 127.0.0.1:49300 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 20:14:01 dist-test-slave-pflr krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
I20260620 20:14:01.104820 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.097429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.24.20.190:38973)
0620 20:14:01.097564 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.097568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.097655 (+ 87us) server_negotiation.cc:407] Connection header received
0620 20:14:01.097733 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.097735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.097769 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.097827 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:01.098411 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.098871 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.099621 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.099816 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.099889 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.102012 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:01.102015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:01.102023 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:01.102044 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:01.103440 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:01.103793 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:01.103796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:01.103797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:01.103867 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:01.104094 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:01.104096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:01.104098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:01.104254 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:01.104353 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:01.104582 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 20:14:01.104710 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:14:01.107823 32180 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:14:01.110752 32180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:14:01.125531 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.118951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:38935 (local address 127.0.0.1:50010)
0620 20:14:01.119794 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:01.119805 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:01.119876 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:01.120613 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:01.120621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:01.120647 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:01.120914 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.120922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.122402 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.122405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.124026 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.124037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.124968 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.124972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.125077 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.125099 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:14:01.125167 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 20:14:01.125340 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":222,"thread_start_us":113,"threads_started":1}
I20260620 20:14:01.125866 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.118782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.24.20.129:44977)
0620 20:14:01.119989 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.120002 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.120038 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:14:01.120270 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.120276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.120484 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.120891 (+ 407us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:01.121938 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.122839 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.123870 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.124563 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.124722 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.124887 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:01.125638 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 20:14:01.125743 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1112,"thread_start_us":74,"threads_started":1}
I20260620 20:14:01.125969 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.119395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.24.20.130:38935)
0620 20:14:01.120068 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.120077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.120118 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:14:01.120195 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.120200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.120404 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.120564 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:01.121052 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.122270 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.124168 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.124829 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.124927 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.125075 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:01.125750 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 20:14:01.125837 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":100,"threads_started":1}
I20260620 20:14:01.126006 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.119517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.24.20.131:33427)
0620 20:14:01.119896 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.119908 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.120227 (+ 319us) server_negotiation.cc:407] Connection header received
0620 20:14:01.120298 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.120302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.120442 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.120568 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:01.121109 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.122264 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.124169 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.124968 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.125066 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.125232 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:01.125603 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 20:14:01.125875 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":113,"threads_started":1}
I20260620 20:14:01.125540 32687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.118601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:44977 (local address 127.0.0.1:47428)
0620 20:14:01.118929 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:01.118980 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:01.119105 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:01.120662 (+ 1557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:01.120664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:01.120676 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:01.120987 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.120993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.122947 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.122953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.123756 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.123765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.124738 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.124742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.124867 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.124924 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 20:14:01.125181 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 20:14:01.125348 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":153,"thread_start_us":93,"threads_started":1}
I20260620 20:14:01.126152 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.119329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:33427 (local address 127.0.0.1:45258)
0620 20:14:01.120026 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:01.120039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:01.120137 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:01.120743 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:01.120746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:01.120760 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:01.120985 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.120993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.122402 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.122405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.124029 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.124037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.125293 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.125296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.125372 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.125393 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:14:01.125463 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:14:01.125521 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":245,"thread_start_us":102,"threads_started":1}
I20260620 20:14:01.128863 32599 tablet_service.cc:1467] Processing CreateTablet for tablet 21902f3c375e4098af795dcad887db94 (DEFAULT_TABLE table=test-table [id=b68315f35d12418c81d84b5dc795fcdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:01.128957 32459 tablet_service.cc:1467] Processing CreateTablet for tablet 21902f3c375e4098af795dcad887db94 (DEFAULT_TABLE table=test-table [id=b68315f35d12418c81d84b5dc795fcdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:01.128958 32319 tablet_service.cc:1467] Processing CreateTablet for tablet 21902f3c375e4098af795dcad887db94 (DEFAULT_TABLE table=test-table [id=b68315f35d12418c81d84b5dc795fcdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:01.129935 32599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21902f3c375e4098af795dcad887db94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:01.129935 32319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21902f3c375e4098af795dcad887db94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:01.129935 32459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21902f3c375e4098af795dcad887db94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:01.136675 32693 tablet_bootstrap.cc:492] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937: Bootstrap starting.
I20260620 20:14:01.138276 32695 tablet_bootstrap.cc:492] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e: Bootstrap starting.
I20260620 20:14:01.138824 32693 tablet_bootstrap.cc:654] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:01.139609 32693 log.cc:824] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:01.140061 32694 tablet_bootstrap.cc:492] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Bootstrap starting.
I20260620 20:14:01.141147 32695 tablet_bootstrap.cc:654] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:01.141454 32693 tablet_bootstrap.cc:492] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937: No bootstrap required, opened a new log
I20260620 20:14:01.141644 32693 ts_tablet_manager.cc:1397] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:14:01.141942 32694 tablet_bootstrap.cc:654] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:01.141956 32695 log.cc:824] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:01.142761 32694 log.cc:824] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:01.143975 32695 tablet_bootstrap.cc:492] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e: No bootstrap required, opened a new log
I20260620 20:14:01.144157 32695 ts_tablet_manager.cc:1397] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:14:01.144415 32694 tablet_bootstrap.cc:492] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: No bootstrap required, opened a new log
I20260620 20:14:01.144584 32694 ts_tablet_manager.cc:1397] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:14:01.145387 32693 raft_consensus.cc:348] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } }
I20260620 20:14:01.145643 32693 raft_consensus.cc:374] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:01.145702 32693 raft_consensus.cc:729] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fadf15df8942e9808a38718ebaa937, State: Initialized, Role: FOLLOWER
I20260620 20:14:01.146173 32693 consensus_queue.cc:260] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } }
I20260620 20:14:01.147140 32693 ts_tablet_manager.cc:1428] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:14:01.147513 32665 heartbeater.cc:503] Master 127.24.20.190:38973 was elected leader, sending a full tablet report...
I20260620 20:14:01.147845 32695 raft_consensus.cc:348] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } }
I20260620 20:14:01.148018 32695 raft_consensus.cc:374] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:01.148048 32695 raft_consensus.cc:729] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa542aa98fb544329ed5ed189e416d0e, State: Initialized, Role: FOLLOWER
I20260620 20:14:01.148120 32694 raft_consensus.cc:348] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } }
I20260620 20:14:01.148346 32694 raft_consensus.cc:374] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:01.148392 32694 raft_consensus.cc:729] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d16d8cf9c7b4dd595d5e1ef9f256e74, State: Initialized, Role: FOLLOWER
I20260620 20:14:01.148387 32695 consensus_queue.cc:260] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } }
I20260620 20:14:01.148824 32694 consensus_queue.cc:260] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } }
I20260620 20:14:01.149255 32695 ts_tablet_manager.cc:1428] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:14:01.149449 32525 heartbeater.cc:503] Master 127.24.20.190:38973 was elected leader, sending a full tablet report...
I20260620 20:14:01.149546 32385 heartbeater.cc:503] Master 127.24.20.190:38973 was elected leader, sending a full tablet report...
I20260620 20:14:01.150578 32694 ts_tablet_manager.cc:1428] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260620 20:14:01.212666 32700 raft_consensus.cc:482] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:14:01.212893 32700 raft_consensus.cc:504] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } }
I20260620 20:14:01.214192 32700 leader_election.cc:288] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7fadf15df8942e9808a38718ebaa937 (127.24.20.131:33427), 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130:38935)
I20260620 20:14:01.219730 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.214569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:41333 (local address 127.24.20.131:33427)
0620 20:14:01.214716 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.214722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.214824 (+ 102us) server_negotiation.cc:407] Connection header received
0620 20:14:01.214975 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.214978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.215028 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.215113 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:01.215554 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.216458 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.218404 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.219025 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.219161 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.219210 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:01.219524 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:14:01.219602 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:14:01.219730 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.214591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:44525 (local address 127.24.20.130:38935)
0620 20:14:01.214777 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.214780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.214815 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:14:01.214975 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.214978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.215028 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.215113 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:01.215554 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.216508 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.218404 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.219042 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.219160 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.219210 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:01.219546 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 20:14:01.219617 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 20:14:01.219820 32702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.214394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:33427 (local address 127.24.20.129:41333)
0620 20:14:01.214733 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:01.214748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:01.214862 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:01.215135 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:01.215138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:01.215163 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:01.215345 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.215350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.216588 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.216594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.218237 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.218248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.219174 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.219178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.219360 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.219412 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 20:14:01.219534 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 20:14:01.219642 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":262,"spinlock_wait_cycles":45568,"thread_start_us":85,"threads_started":1}
I20260620 20:14:01.219831 32703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.214394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:38935 (local address 127.24.20.129:44525)
0620 20:14:01.214733 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:01.214748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:01.214862 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:01.215135 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:01.215138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:01.215163 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:01.215340 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.215348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.216637 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.216640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.218237 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.218248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.219174 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.219178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.219355 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.219413 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 20:14:01.219530 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 20:14:01.219641 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":350,"spinlock_wait_cycles":26880,"thread_start_us":98,"threads_started":1}
I20260620 20:14:01.220463 32619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21902f3c375e4098af795dcad887db94" candidate_uuid: "fa542aa98fb544329ed5ed189e416d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fadf15df8942e9808a38718ebaa937" is_pre_election: true
I20260620 20:14:01.220642 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21902f3c375e4098af795dcad887db94" candidate_uuid: "fa542aa98fb544329ed5ed189e416d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" is_pre_election: true
I20260620 20:14:01.220822 32619 raft_consensus.cc:2454] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa542aa98fb544329ed5ed189e416d0e in term 0.
I20260620 20:14:01.220880 32479 raft_consensus.cc:2454] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa542aa98fb544329ed5ed189e416d0e in term 0.
I20260620 20:14:01.221321 32270 leader_election.cc:302] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9d16d8cf9c7b4dd595d5e1ef9f256e74, f7fadf15df8942e9808a38718ebaa937, fa542aa98fb544329ed5ed189e416d0e; no voters:
I20260620 20:14:01.221592 32700 raft_consensus.cc:2781] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:14:01.221685 32700 raft_consensus.cc:482] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:14:01.221746 32700 raft_consensus.cc:3037] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:01.222777 32700 raft_consensus.cc:504] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } }
I20260620 20:14:01.223106 32700 leader_election.cc:288] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [CANDIDATE]: Term 1 election: Requested vote from peers f7fadf15df8942e9808a38718ebaa937 (127.24.20.131:33427), 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130:38935)
I20260620 20:14:01.223423 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21902f3c375e4098af795dcad887db94" candidate_uuid: "fa542aa98fb544329ed5ed189e416d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74"
I20260620 20:14:01.223555 32479 raft_consensus.cc:3037] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:01.224537 32479 raft_consensus.cc:2454] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa542aa98fb544329ed5ed189e416d0e in term 1.
I20260620 20:14:01.224831 32619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21902f3c375e4098af795dcad887db94" candidate_uuid: "fa542aa98fb544329ed5ed189e416d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fadf15df8942e9808a38718ebaa937"
I20260620 20:14:01.224879 32270 leader_election.cc:302] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d16d8cf9c7b4dd595d5e1ef9f256e74, fa542aa98fb544329ed5ed189e416d0e; no voters:
I20260620 20:14:01.224967 32619 raft_consensus.cc:3037] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:01.225068 32700 raft_consensus.cc:2781] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:01.225291 32700 raft_consensus.cc:686] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [term 1 LEADER]: Becoming Leader. State: Replica: fa542aa98fb544329ed5ed189e416d0e, State: Running, Role: LEADER
I20260620 20:14:01.225649 32700 consensus_queue.cc:237] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } }
I20260620 20:14:01.226524 32619 raft_consensus.cc:2454] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa542aa98fb544329ed5ed189e416d0e in term 1.
I20260620 20:14:01.229104 32180 catalog_manager.cc:5495] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e reported cstate change: term changed from 0 to 1, leader changed from <none> to fa542aa98fb544329ed5ed189e416d0e (127.24.20.129). New cstate: current_term: 1 leader_uuid: "fa542aa98fb544329ed5ed189e416d0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } health_report { overall_health: UNKNOWN } } }
I20260620 20:14:01.245359 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.241816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.24.20.129:44977)
0620 20:14:01.241928 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.241937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.241953 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:14:01.242110 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.242113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.242166 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.242254 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:01.242477 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.243047 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.244041 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.244243 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.244352 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.244547 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:01.244995 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:01.245092 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:01.245210 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 20:14:01.245255 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:14:01.251251 32479 raft_consensus.cc:1261] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Refusing update from remote peer fa542aa98fb544329ed5ed189e416d0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:01.251291 32619 raft_consensus.cc:1261] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Refusing update from remote peer fa542aa98fb544329ed5ed189e416d0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:01.251953 32704 consensus_queue.cc:1035] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:01.252166 32700 consensus_queue.cc:1035] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:01.260757 32709 mvcc.cc:205] Tried to move back new op lower bound from 7299016463358201856 to 7299016463263531008. Current Snapshot: MvccSnapshot[applied={T|T < 7299016463358201856}]
I20260620 20:14:01.268926 32182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:01.269097 32182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:01.271515 32182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995: Sending DeleteTablet for 3 replicas of tablet 21902f3c375e4098af795dcad887db94
I20260620 20:14:01.272300 32459 tablet_service.cc:1514] Processing DeleteTablet for tablet 21902f3c375e4098af795dcad887db94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:01 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:50010
I20260620 20:14:01.272292 32599 tablet_service.cc:1514] Processing DeleteTablet for tablet 21902f3c375e4098af795dcad887db94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:01 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:45258
I20260620 20:14:01.272812 32319 tablet_service.cc:1514] Processing DeleteTablet for tablet 21902f3c375e4098af795dcad887db94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:01 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:47428
I20260620 20:14:01.273072 32182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:14:01.273408 32182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:14:01.273439 32718 tablet_replica.cc:330] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e: stopping tablet replica
I20260620 20:14:01.273846 32718 raft_consensus.cc:2229] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:14:01.274075 32717 tablet_replica.cc:330] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937: stopping tablet replica
I20260620 20:14:01.274330 32717 raft_consensus.cc:2229] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:01.274343 32718 raft_consensus.cc:2258] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:01.274621 32717 raft_consensus.cc:2258] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:01.274895 32716 tablet_replica.cc:330] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: stopping tablet replica
I20260620 20:14:01.275141 32716 raft_consensus.cc:2229] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:01.275414 32716 raft_consensus.cc:2258] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:01.277603 32716 ts_tablet_manager.cc:1905] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:01.277823 32717 ts_tablet_manager.cc:1905] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:01.278555 32319 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3e8f62e2a94c6a870f5d6e88fb59ee (DEFAULT_TABLE table=test-table [id=78217c822a7a42a8a98d205a63ae0782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:01.278816 32319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c3e8f62e2a94c6a870f5d6e88fb59ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:01.280615 32717 ts_tablet_manager.cc:1918] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:01.280767 32717 log.cc:1196] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/21902f3c375e4098af795dcad887db94
I20260620 20:14:01.281131 32717 ts_tablet_manager.cc:1939] T 21902f3c375e4098af795dcad887db94 P f7fadf15df8942e9808a38718ebaa937: Deleting consensus metadata
I20260620 20:14:01.281435 32599 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3e8f62e2a94c6a870f5d6e88fb59ee (DEFAULT_TABLE table=test-table [id=78217c822a7a42a8a98d205a63ae0782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:01.281589 32716 ts_tablet_manager.cc:1918] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:01.281682 32716 log.cc:1196] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/21902f3c375e4098af795dcad887db94
I20260620 20:14:01.281689 32599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c3e8f62e2a94c6a870f5d6e88fb59ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:01.281983 32716 ts_tablet_manager.cc:1939] T 21902f3c375e4098af795dcad887db94 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Deleting consensus metadata
I20260620 20:14:01.283753 32166 catalog_manager.cc:4841] TS 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130:38935): tablet 21902f3c375e4098af795dcad887db94 (table test-table [id=b68315f35d12418c81d84b5dc795fcdb]) successfully deleted
I20260620 20:14:01.283816 32693 tablet_bootstrap.cc:492] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937: Bootstrap starting.
I20260620 20:14:01.283881 32695 tablet_bootstrap.cc:492] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e: Bootstrap starting.
I20260620 20:14:01.284570 32693 tablet_bootstrap.cc:654] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:01.284646 32695 tablet_bootstrap.cc:654] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:01.285316 32459 tablet_service.cc:1467] Processing CreateTablet for tablet 9c3e8f62e2a94c6a870f5d6e88fb59ee (DEFAULT_TABLE table=test-table [id=78217c822a7a42a8a98d205a63ae0782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:01.285552 32459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c3e8f62e2a94c6a870f5d6e88fb59ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:01.285769 32695 tablet_bootstrap.cc:492] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e: No bootstrap required, opened a new log
I20260620 20:14:01.285837 32695 ts_tablet_manager.cc:1397] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 20:14:01.285992 32693 tablet_bootstrap.cc:492] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937: No bootstrap required, opened a new log
I20260620 20:14:01.286067 32693 ts_tablet_manager.cc:1397] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:14:01.286233 32695 raft_consensus.cc:348] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.286329 32695 raft_consensus.cc:374] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:01.286357 32695 raft_consensus.cc:729] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa542aa98fb544329ed5ed189e416d0e, State: Initialized, Role: FOLLOWER
I20260620 20:14:01.286459 32695 consensus_queue.cc:260] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.286463 32693 raft_consensus.cc:348] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.286583 32693 raft_consensus.cc:374] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:01.286616 32693 raft_consensus.cc:729] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fadf15df8942e9808a38718ebaa937, State: Initialized, Role: FOLLOWER
I20260620 20:14:01.286710 32695 ts_tablet_manager.cc:1428] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 20:14:01.286736 32693 consensus_queue.cc:260] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.286969 32693 ts_tablet_manager.cc:1428] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:14:01.287645 32694 tablet_bootstrap.cc:492] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Bootstrap starting.
I20260620 20:14:01.288195 32168 catalog_manager.cc:4841] TS f7fadf15df8942e9808a38718ebaa937 (127.24.20.131:33427): tablet 21902f3c375e4098af795dcad887db94 (table test-table [id=b68315f35d12418c81d84b5dc795fcdb]) successfully deleted
I20260620 20:14:01.288527 32694 tablet_bootstrap.cc:654] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:01.289476 32718 ts_tablet_manager.cc:1905] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:01.291378 32718 ts_tablet_manager.cc:1918] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:01.291477 32718 log.cc:1196] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/wal/wals/21902f3c375e4098af795dcad887db94
I20260620 20:14:01.291780 32718 ts_tablet_manager.cc:1939] T 21902f3c375e4098af795dcad887db94 P fa542aa98fb544329ed5ed189e416d0e: Deleting consensus metadata
I20260620 20:14:01.292865 32168 catalog_manager.cc:4841] TS fa542aa98fb544329ed5ed189e416d0e (127.24.20.129:44977): tablet 21902f3c375e4098af795dcad887db94 (table test-table [id=b68315f35d12418c81d84b5dc795fcdb]) successfully deleted
I20260620 20:14:01.293732 32694 tablet_bootstrap.cc:492] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74: No bootstrap required, opened a new log
I20260620 20:14:01.293810 32694 ts_tablet_manager.cc:1397] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 20:14:01.294210 32694 raft_consensus.cc:348] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.294303 32694 raft_consensus.cc:374] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:01.294332 32694 raft_consensus.cc:729] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d16d8cf9c7b4dd595d5e1ef9f256e74, State: Initialized, Role: FOLLOWER
I20260620 20:14:01.294448 32694 consensus_queue.cc:260] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.294690 32694 ts_tablet_manager.cc:1428] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260620 20:14:01.307303 32666 tablet.cc:2369] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:14:01.347836 32526 tablet.cc:2369] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:14:01.368458 32704 raft_consensus.cc:482] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:14:01.368606 32704 raft_consensus.cc:504] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.369048 32704 leader_election.cc:288] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130:38935), f7fadf15df8942e9808a38718ebaa937 (127.24.20.131:33427)
I20260620 20:14:01.369437 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3e8f62e2a94c6a870f5d6e88fb59ee" candidate_uuid: "fa542aa98fb544329ed5ed189e416d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" is_pre_election: true
I20260620 20:14:01.369582 32479 raft_consensus.cc:2454] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa542aa98fb544329ed5ed189e416d0e in term 0.
I20260620 20:14:01.369602 32619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3e8f62e2a94c6a870f5d6e88fb59ee" candidate_uuid: "fa542aa98fb544329ed5ed189e416d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fadf15df8942e9808a38718ebaa937" is_pre_election: true
I20260620 20:14:01.369755 32619 raft_consensus.cc:2454] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa542aa98fb544329ed5ed189e416d0e in term 0.
I20260620 20:14:01.369874 32270 leader_election.cc:302] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d16d8cf9c7b4dd595d5e1ef9f256e74, fa542aa98fb544329ed5ed189e416d0e; no voters:
I20260620 20:14:01.370044 32704 raft_consensus.cc:2781] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:14:01.370083 32704 raft_consensus.cc:482] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:14:01.370117 32704 raft_consensus.cc:3037] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:01.371316 32704 raft_consensus.cc:504] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.371755 32704 leader_election.cc:288] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [CANDIDATE]: Term 1 election: Requested vote from peers 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130:38935), f7fadf15df8942e9808a38718ebaa937 (127.24.20.131:33427)
I20260620 20:14:01.372161 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3e8f62e2a94c6a870f5d6e88fb59ee" candidate_uuid: "fa542aa98fb544329ed5ed189e416d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74"
I20260620 20:14:01.372227 32619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c3e8f62e2a94c6a870f5d6e88fb59ee" candidate_uuid: "fa542aa98fb544329ed5ed189e416d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fadf15df8942e9808a38718ebaa937"
I20260620 20:14:01.372284 32479 raft_consensus.cc:3037] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:01.372329 32619 raft_consensus.cc:3037] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:01.373456 32619 raft_consensus.cc:2454] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa542aa98fb544329ed5ed189e416d0e in term 1.
I20260620 20:14:01.373452 32479 raft_consensus.cc:2454] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa542aa98fb544329ed5ed189e416d0e in term 1.
I20260620 20:14:01.373767 32272 leader_election.cc:302] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f7fadf15df8942e9808a38718ebaa937, fa542aa98fb544329ed5ed189e416d0e; no voters:
I20260620 20:14:01.373957 32704 raft_consensus.cc:2781] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:01.374029 32704 raft_consensus.cc:686] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [term 1 LEADER]: Becoming Leader. State: Replica: fa542aa98fb544329ed5ed189e416d0e, State: Running, Role: LEADER
I20260620 20:14:01.374145 32704 consensus_queue.cc:237] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.375686 32182 catalog_manager.cc:5495] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e reported cstate change: term changed from 0 to 1, leader changed from <none> to fa542aa98fb544329ed5ed189e416d0e (127.24.20.129). New cstate: current_term: 1 leader_uuid: "fa542aa98fb544329ed5ed189e416d0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } health_report { overall_health: UNKNOWN } } }
W20260620 20:14:01.389993 32386 tablet.cc:2369] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:14:01 dist-test-slave-pflr krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986440, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.190@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
I20260620 20:14:01.430876 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.422617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:38973 (local address 127.0.0.1:49322)
0620 20:14:01.422915 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:01.422929 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:01.423058 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:01.423301 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:01.423304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:01.423771 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:01.423893 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.423901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.424727 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.424731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.425429 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.425438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.425863 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.425865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.425924 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.426459 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:01.426471 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:01.427863 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:01.429486 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:01.429494 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:01.429498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:01.429713 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:01.429934 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:01.429936 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:01.429937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:01.430008 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:01.430310 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:01.430315 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:01.430406 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 20:14:01.430495 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 20:14:01.430741 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
I20260620 20:14:01.430962 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.422637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.24.20.190:38973)
0620 20:14:01.422768 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.422773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.423010 (+ 237us) server_negotiation.cc:407] Connection header received
0620 20:14:01.423145 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.423148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.423187 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.423255 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:01.424017 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.424583 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.425559 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.425765 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.425845 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.427979 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:01.427982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:01.427984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:01.428004 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:01.429402 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:01.429792 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:01.429794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:01.429796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:01.429863 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:01.430079 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:01.430081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:01.430082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:01.430228 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:01.430299 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:01.430635 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 20:14:01.430828 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":49}
I20260620 20:14:01.434677 32182 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:49322:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:14:01.435391 32182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:14:01.439921 32319 tablet_service.cc:1467] Processing CreateTablet for tablet d5b23c6d21cc468a950859c15a7c870e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d240f578f19a4935afed172dea828408]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:14:01.439946 32459 tablet_service.cc:1467] Processing CreateTablet for tablet d5b23c6d21cc468a950859c15a7c870e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d240f578f19a4935afed172dea828408]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:14:01.439977 32599 tablet_service.cc:1467] Processing CreateTablet for tablet d5b23c6d21cc468a950859c15a7c870e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d240f578f19a4935afed172dea828408]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:14:01.440305 32459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b23c6d21cc468a950859c15a7c870e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:01.440304 32599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b23c6d21cc468a950859c15a7c870e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:01.440320 32319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b23c6d21cc468a950859c15a7c870e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:01.442533 32695 tablet_bootstrap.cc:492] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e: Bootstrap starting.
I20260620 20:14:01.443118 32693 tablet_bootstrap.cc:492] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937: Bootstrap starting.
I20260620 20:14:01.443230 32694 tablet_bootstrap.cc:492] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Bootstrap starting.
I20260620 20:14:01.443421 32695 tablet_bootstrap.cc:654] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:01.444097 32694 tablet_bootstrap.cc:654] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:01.444129 32693 tablet_bootstrap.cc:654] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:01.444725 32695 tablet_bootstrap.cc:492] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e: No bootstrap required, opened a new log
I20260620 20:14:01.444815 32695 ts_tablet_manager.cc:1397] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 20:14:01.445217 32695 raft_consensus.cc:348] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.445331 32695 raft_consensus.cc:374] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:01.445363 32694 tablet_bootstrap.cc:492] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74: No bootstrap required, opened a new log
I20260620 20:14:01.445380 32695 raft_consensus.cc:729] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa542aa98fb544329ed5ed189e416d0e, State: Initialized, Role: FOLLOWER
I20260620 20:14:01.445442 32694 ts_tablet_manager.cc:1397] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 20:14:01.445456 32693 tablet_bootstrap.cc:492] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937: No bootstrap required, opened a new log
I20260620 20:14:01.445538 32693 ts_tablet_manager.cc:1397] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 20:14:01.445526 32695 consensus_queue.cc:260] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.445802 32695 ts_tablet_manager.cc:1428] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:14:01.445869 32694 raft_consensus.cc:348] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.445984 32694 raft_consensus.cc:374] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:01.445935 32693 raft_consensus.cc:348] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.446015 32694 raft_consensus.cc:729] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d16d8cf9c7b4dd595d5e1ef9f256e74, State: Initialized, Role: FOLLOWER
I20260620 20:14:01.446027 32693 raft_consensus.cc:374] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:01.445996 32704 tablet_replica.cc:439] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } }
I20260620 20:14:01.446061 32693 raft_consensus.cc:729] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fadf15df8942e9808a38718ebaa937, State: Initialized, Role: FOLLOWER
I20260620 20:14:01.446110 32704 tablet_replica.cc:442] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:14:01.446138 32694 consensus_queue.cc:260] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.446178 32693 consensus_queue.cc:260] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.446424 32693 ts_tablet_manager.cc:1428] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 20:14:01.446401 32694 ts_tablet_manager.cc:1428] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 20:14:01.446378 32701 tablet_replica.cc:439] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } }
I20260620 20:14:01.446555 32701 tablet_replica.cc:442] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:14:01.446488 32699 tablet_replica.cc:439] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } }
I20260620 20:14:01.446614 32699 tablet_replica.cc:442] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:14:01.763075 32704 consensus_queue.cc:1035] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:01.766383 32704 consensus_queue.cc:1035] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:01.852630 32704 raft_consensus.cc:482] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:14:01.852806 32704 raft_consensus.cc:504] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.853256 32704 leader_election.cc:288] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130:38935), f7fadf15df8942e9808a38718ebaa937 (127.24.20.131:33427)
I20260620 20:14:01.853639 32619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b23c6d21cc468a950859c15a7c870e" candidate_uuid: "fa542aa98fb544329ed5ed189e416d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fadf15df8942e9808a38718ebaa937" is_pre_election: true
I20260620 20:14:01.853660 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b23c6d21cc468a950859c15a7c870e" candidate_uuid: "fa542aa98fb544329ed5ed189e416d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" is_pre_election: true
I20260620 20:14:01.853799 32479 raft_consensus.cc:2454] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa542aa98fb544329ed5ed189e416d0e in term 0.
I20260620 20:14:01.853796 32619 raft_consensus.cc:2454] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa542aa98fb544329ed5ed189e416d0e in term 0.
I20260620 20:14:01.854115 32270 leader_election.cc:302] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d16d8cf9c7b4dd595d5e1ef9f256e74, fa542aa98fb544329ed5ed189e416d0e; no voters:
I20260620 20:14:01.854326 32704 raft_consensus.cc:2781] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:14:01.854396 32704 raft_consensus.cc:482] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:14:01.854452 32704 raft_consensus.cc:3037] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:01.855400 32704 raft_consensus.cc:504] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.855729 32704 leader_election.cc:288] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [CANDIDATE]: Term 1 election: Requested vote from peers 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130:38935), f7fadf15df8942e9808a38718ebaa937 (127.24.20.131:33427)
I20260620 20:14:01.855997 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b23c6d21cc468a950859c15a7c870e" candidate_uuid: "fa542aa98fb544329ed5ed189e416d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74"
I20260620 20:14:01.856060 32619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b23c6d21cc468a950859c15a7c870e" candidate_uuid: "fa542aa98fb544329ed5ed189e416d0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fadf15df8942e9808a38718ebaa937"
I20260620 20:14:01.856164 32619 raft_consensus.cc:3037] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:01.856165 32479 raft_consensus.cc:3037] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:01.857072 32479 raft_consensus.cc:2454] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa542aa98fb544329ed5ed189e416d0e in term 1.
I20260620 20:14:01.857264 32619 raft_consensus.cc:2454] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa542aa98fb544329ed5ed189e416d0e in term 1.
I20260620 20:14:01.857350 32270 leader_election.cc:302] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d16d8cf9c7b4dd595d5e1ef9f256e74, fa542aa98fb544329ed5ed189e416d0e; no voters:
I20260620 20:14:01.857515 32704 raft_consensus.cc:2781] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:01.857599 32704 raft_consensus.cc:686] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [term 1 LEADER]: Becoming Leader. State: Replica: fa542aa98fb544329ed5ed189e416d0e, State: Running, Role: LEADER
I20260620 20:14:01.857757 32704 consensus_queue.cc:237] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:01.858395 32704 tablet_replica.cc:439] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e: TxnStatusTablet state changed. Reason: New leader fa542aa98fb544329ed5ed189e416d0e. Latest consensus state: current_term: 1 leader_uuid: "fa542aa98fb544329ed5ed189e416d0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } }
I20260620 20:14:01.858490 32704 tablet_replica.cc:442] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:14:01.858848 32739 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:14:01.859017 32739 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:14:01.859505 32182 catalog_manager.cc:5495] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e reported cstate change: term changed from 0 to 1, leader changed from <none> to fa542aa98fb544329ed5ed189e416d0e (127.24.20.129). New cstate: current_term: 1 leader_uuid: "fa542aa98fb544329ed5ed189e416d0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } health_report { overall_health: UNKNOWN } } }
I20260620 20:14:01.899746 32728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.896252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:44977 (local address 127.0.0.1:47454)
0620 20:14:01.896419 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:01.896437 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:01.896549 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:01.897063 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:01.897066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:01.897074 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:14:01.897190 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.897196 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.897927 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.897933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.898536 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.898545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.899011 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.899015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.899100 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.899143 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:14:01.899437 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:14:01.899455 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 20:14:01.899538 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 20:14:01.899617 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 20:14:01.899751 32742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.896346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.24.20.129:44977)
0620 20:14:01.896849 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.896852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.896864 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:14:01.896904 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.896906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.896947 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.897041 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:01.897310 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.897823 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.898655 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.898897 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.899000 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.899286 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:01.899359 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:01.899433 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:01.899551 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 20:14:01.899602 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":62,"threads_started":1}
I20260620 20:14:01.902570 32479 raft_consensus.cc:1261] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Refusing update from remote peer fa542aa98fb544329ed5ed189e416d0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:01.902591 32619 raft_consensus.cc:1261] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Refusing update from remote peer fa542aa98fb544329ed5ed189e416d0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:01.903049 32734 consensus_queue.cc:1035] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:01.903185 32704 consensus_queue.cc:1035] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:01.905827 32699 tablet_replica.cc:439] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937: TxnStatusTablet state changed. Reason: New leader fa542aa98fb544329ed5ed189e416d0e. Latest consensus state: current_term: 1 leader_uuid: "fa542aa98fb544329ed5ed189e416d0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } }
I20260620 20:14:01.905951 32699 tablet_replica.cc:442] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:14:01.906200 32701 tablet_replica.cc:439] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74: TxnStatusTablet state changed. Reason: New leader fa542aa98fb544329ed5ed189e416d0e. Latest consensus state: current_term: 1 leader_uuid: "fa542aa98fb544329ed5ed189e416d0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } }
I20260620 20:14:01.906301 32701 tablet_replica.cc:442] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:14:01.907003 32700 tablet_replica.cc:439] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa542aa98fb544329ed5ed189e416d0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } }
I20260620 20:14:01.907105 32700 tablet_replica.cc:442] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:14:01.907363 32737 mvcc.cc:205] Tried to move back new op lower bound from 7299016466028683264 to 7299016465850691584. Current Snapshot: MvccSnapshot[applied={T|T < 7299016466028683264}]
I20260620 20:14:01.907542 32699 tablet_replica.cc:439] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa542aa98fb544329ed5ed189e416d0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } }
I20260620 20:14:01.907635 32699 tablet_replica.cc:442] T d5b23c6d21cc468a950859c15a7c870e P f7fadf15df8942e9808a38718ebaa937: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:14:01.907984 32700 tablet_replica.cc:439] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa542aa98fb544329ed5ed189e416d0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } }
I20260620 20:14:01.908069 32700 tablet_replica.cc:442] T d5b23c6d21cc468a950859c15a7c870e P fa542aa98fb544329ed5ed189e416d0e: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:14:01.908377 32733 mvcc.cc:205] Tried to move back new op lower bound from 7299016466028683264 to 7299016465850691584. Current Snapshot: MvccSnapshot[applied={T|T < 7299016466028683264}]
I20260620 20:14:01.910336 32701 tablet_replica.cc:439] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa542aa98fb544329ed5ed189e416d0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } } }
I20260620 20:14:01.910435 32701 tablet_replica.cc:442] T d5b23c6d21cc468a950859c15a7c870e P 9d16d8cf9c7b4dd595d5e1ef9f256e74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:14:01.922031 32750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.918447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:44977 (local address 127.24.20.129:55777)
0620 20:14:01.918745 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:01.918760 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:01.918860 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:01.919063 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:01.919066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:01.919073 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:14:01.919217 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.919223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.919883 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.919887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.920869 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:01.920880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.921301 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.921304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:01.921391 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.921438 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:14:01.921742 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:14:01.921745 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 20:14:01.921814 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 20:14:01.921882 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":16,"thread_start_us":104,"threads_started":1}
I20260620 20:14:01.922031 32742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.918492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:55777 (local address 127.24.20.129:44977)
0620 20:14:01.918613 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.918618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.918830 (+ 212us) server_negotiation.cc:407] Connection header received
0620 20:14:01.918937 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.918939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.918977 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.919037 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:01.919302 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.919784 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.920974 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.921203 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.921280 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.921565 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:01.921647 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:01.921711 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:01.921826 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 20:14:01.921871 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":45}
I20260620 20:14:01.958679 32182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:01.958850 32182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:01.960819 32182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995: Sending DeleteTablet for 3 replicas of tablet 9c3e8f62e2a94c6a870f5d6e88fb59ee
I20260620 20:14:01.961354 32319 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c3e8f62e2a94c6a870f5d6e88fb59ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:01 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:47428
I20260620 20:14:01.961455 32459 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c3e8f62e2a94c6a870f5d6e88fb59ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:01 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:50010
I20260620 20:14:01.961707 32599 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c3e8f62e2a94c6a870f5d6e88fb59ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:01 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:45258
I20260620 20:14:01.961975 32755 tablet_replica.cc:330] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937: stopping tablet replica
I20260620 20:14:01.962109 32754 tablet_replica.cc:330] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74: stopping tablet replica
I20260620 20:14:01.962186 32755 raft_consensus.cc:2229] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:01.962222 32754 raft_consensus.cc:2229] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:01.962329 32754 raft_consensus.cc:2258] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:01.962365 32755 raft_consensus.cc:2258] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:01.962924 32754 ts_tablet_manager.cc:1905] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:01.964741 32753 tablet_replica.cc:330] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e: stopping tablet replica
I20260620 20:14:01.964896 32753 raft_consensus.cc:2229] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:14:01.965126 32753 raft_consensus.cc:2258] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:01.965152 32755 ts_tablet_manager.cc:1905] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:01.965457 32754 ts_tablet_manager.cc:1918] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:14:01.965536 32754 log.cc:1196] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/9c3e8f62e2a94c6a870f5d6e88fb59ee
I20260620 20:14:01.965802 32754 ts_tablet_manager.cc:1939] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Deleting consensus metadata
I20260620 20:14:01.966041 32753 ts_tablet_manager.cc:1905] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:01.966449 32166 catalog_manager.cc:4841] TS 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130:38935): tablet 9c3e8f62e2a94c6a870f5d6e88fb59ee (table test-table [id=78217c822a7a42a8a98d205a63ae0782]) successfully deleted
I20260620 20:14:01.967742 32755 ts_tablet_manager.cc:1918] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:14:01.967832 32755 log.cc:1196] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/9c3e8f62e2a94c6a870f5d6e88fb59ee
I20260620 20:14:01.968077 32755 ts_tablet_manager.cc:1939] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P f7fadf15df8942e9808a38718ebaa937: Deleting consensus metadata
I20260620 20:14:01.968333 32753 ts_tablet_manager.cc:1918] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:14:01.968418 32753 log.cc:1196] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/wal/wals/9c3e8f62e2a94c6a870f5d6e88fb59ee
I20260620 20:14:01.968700 32168 catalog_manager.cc:4841] TS f7fadf15df8942e9808a38718ebaa937 (127.24.20.131:33427): tablet 9c3e8f62e2a94c6a870f5d6e88fb59ee (table test-table [id=78217c822a7a42a8a98d205a63ae0782]) successfully deleted
I20260620 20:14:01.968743 32753 ts_tablet_manager.cc:1939] T 9c3e8f62e2a94c6a870f5d6e88fb59ee P fa542aa98fb544329ed5ed189e416d0e: Deleting consensus metadata
I20260620 20:14:01.969272 32168 catalog_manager.cc:4841] TS fa542aa98fb544329ed5ed189e416d0e (127.24.20.129:44977): tablet 9c3e8f62e2a94c6a870f5d6e88fb59ee (table test-table [id=78217c822a7a42a8a98d205a63ae0782]) successfully deleted
Jun 20 20:14:01 dist-test-slave-pflr krb5kdc[32137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:14:01 dist-test-slave-pflr krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:01 dist-test-slave-pflr krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:14:01.991597 32763 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:01.984922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.24.20.190:38973)
0620 20:14:01.985176 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.985181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.985199 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:14:01.985341 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.985345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.985401 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.985507 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:01.986163 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.986995 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.987844 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.988196 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.988310 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:01.991457 (+ 3147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49330: BlockingRecv error: recv got EOF from 127.0.0.1:49330 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
Jun 20 20:14:01 dist-test-slave-pflr krb5kdc[32137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM
I20260620 20:14:02.004688 32763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:01.992953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.24.20.190:38973)
0620 20:14:01.993054 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:01.993058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:01.996428 (+ 3370us) server_negotiation.cc:407] Connection header received
0620 20:14:01.996498 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:01.996501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:01.996543 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:01.996648 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:01.997183 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.997936 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.998676 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:01.999008 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:01.999107 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.001230 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:02.001234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:02.001237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:02.001262 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:02.002969 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:02.003373 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:02.003377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:02.003379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:02.003464 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:02.003789 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:02.003792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:02.003794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:02.004047 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:02.004174 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:02.004372 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 20:14:02.004486 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:14:02.007460 32182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:14:02.007923 32182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:14:02.012365 32319 tablet_service.cc:1467] Processing CreateTablet for tablet b7373d4fb2dc4d4ca373dffba82b7a82 (DEFAULT_TABLE table=test-table [id=ca7870b571a54b288aee3c240c2c310d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:02.012689 32319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7373d4fb2dc4d4ca373dffba82b7a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:02.014935 301 tablet_bootstrap.cc:492] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e: Bootstrap starting.
I20260620 20:14:02.014914 32599 tablet_service.cc:1467] Processing CreateTablet for tablet b7373d4fb2dc4d4ca373dffba82b7a82 (DEFAULT_TABLE table=test-table [id=ca7870b571a54b288aee3c240c2c310d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:02.015175 32599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7373d4fb2dc4d4ca373dffba82b7a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:02.015748 301 tablet_bootstrap.cc:654] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:02.016881 301 tablet_bootstrap.cc:492] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e: No bootstrap required, opened a new log
I20260620 20:14:02.016885 32459 tablet_service.cc:1467] Processing CreateTablet for tablet b7373d4fb2dc4d4ca373dffba82b7a82 (DEFAULT_TABLE table=test-table [id=ca7870b571a54b288aee3c240c2c310d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:02.016976 301 ts_tablet_manager.cc:1397] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:14:02.017119 32459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7373d4fb2dc4d4ca373dffba82b7a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:02.017380 301 raft_consensus.cc:348] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.017513 301 raft_consensus.cc:374] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:02.017546 301 raft_consensus.cc:729] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa542aa98fb544329ed5ed189e416d0e, State: Initialized, Role: FOLLOWER
I20260620 20:14:02.017741 303 tablet_bootstrap.cc:492] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937: Bootstrap starting.
I20260620 20:14:02.017920 301 consensus_queue.cc:260] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.018654 301 ts_tablet_manager.cc:1428] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:14:02.018692 303 tablet_bootstrap.cc:654] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:02.019562 305 tablet_bootstrap.cc:492] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Bootstrap starting.
I20260620 20:14:02.019901 303 tablet_bootstrap.cc:492] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937: No bootstrap required, opened a new log
I20260620 20:14:02.019976 303 ts_tablet_manager.cc:1397] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:14:02.020466 305 tablet_bootstrap.cc:654] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:02.020517 303 raft_consensus.cc:348] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.020663 303 raft_consensus.cc:374] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:02.020701 303 raft_consensus.cc:729] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fadf15df8942e9808a38718ebaa937, State: Initialized, Role: FOLLOWER
I20260620 20:14:02.020826 303 consensus_queue.cc:260] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.021505 303 ts_tablet_manager.cc:1428] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:14:02.021826 305 tablet_bootstrap.cc:492] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: No bootstrap required, opened a new log
I20260620 20:14:02.021914 305 ts_tablet_manager.cc:1397] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 20:14:02.022444 305 raft_consensus.cc:348] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.022552 305 raft_consensus.cc:374] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:02.022589 305 raft_consensus.cc:729] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d16d8cf9c7b4dd595d5e1ef9f256e74, State: Initialized, Role: FOLLOWER
I20260620 20:14:02.022722 305 consensus_queue.cc:260] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.023010 305 ts_tablet_manager.cc:1428] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 20:14:02.197933 32701 raft_consensus.cc:482] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:14:02.198066 32701 raft_consensus.cc:504] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.199185 32701 leader_election.cc:288] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa542aa98fb544329ed5ed189e416d0e (127.24.20.129:44977), f7fadf15df8942e9808a38718ebaa937 (127.24.20.131:33427)
I20260620 20:14:02.203951 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.199472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:44977 (local address 127.24.20.130:33965)
0620 20:14:02.199803 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:02.199816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:02.199958 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:02.200381 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:02.200384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:02.200511 (+ 127us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:02.200683 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:02.200691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.201617 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.201620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:02.202543 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:02.202552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.203411 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.203414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:02.203560 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.203586 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 20:14:02.203687 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:14:02.203758 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":24,"spinlock_wait_cycles":2944,"thread_start_us":140,"threads_started":1}
I20260620 20:14:02.203955 32742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.199538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:33965 (local address 127.24.20.129:44977)
0620 20:14:02.199697 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:02.199702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:02.199910 (+ 208us) server_negotiation.cc:407] Connection header received
0620 20:14:02.200102 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:02.200106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:02.200165 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:02.200221 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:02.200834 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.201457 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.202657 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.203296 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.203388 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.203465 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:02.203730 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 20:14:02.203816 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:14:02.204471 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.199957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:33427 (local address 127.24.20.130:33633)
0620 20:14:02.200300 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:02.200318 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:02.200439 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:02.200764 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:02.200767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:02.200782 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:02.201304 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:02.201313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.202065 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.202068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:02.203082 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:02.203092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.203925 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.203929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:02.204050 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.204077 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 20:14:02.204210 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 20:14:02.204287 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":2,"spinlock_wait_cycles":2432,"thread_start_us":62,"threads_started":1}
I20260620 20:14:02.204540 32339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7373d4fb2dc4d4ca373dffba82b7a82" candidate_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa542aa98fb544329ed5ed189e416d0e" is_pre_election: true
I20260620 20:14:02.204833 32339 raft_consensus.cc:2454] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d16d8cf9c7b4dd595d5e1ef9f256e74 in term 0.
I20260620 20:14:02.205178 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.200064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:33633 (local address 127.24.20.131:33427)
0620 20:14:02.200332 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:02.200337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:02.200381 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:14:02.200569 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:02.200573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:02.200651 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:02.200744 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:02.201420 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.201936 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.203208 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.203795 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.203911 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.203954 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:02.204270 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 20:14:02.205072 (+ 802us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
I20260620 20:14:02.205267 32412 leader_election.cc:302] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d16d8cf9c7b4dd595d5e1ef9f256e74, fa542aa98fb544329ed5ed189e416d0e; no voters:
I20260620 20:14:02.205559 32701 raft_consensus.cc:2781] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:14:02.205632 32701 raft_consensus.cc:482] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:14:02.205576 32619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7373d4fb2dc4d4ca373dffba82b7a82" candidate_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fadf15df8942e9808a38718ebaa937" is_pre_election: true
I20260620 20:14:02.205685 32701 raft_consensus.cc:3037] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:02.205731 32619 raft_consensus.cc:2454] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d16d8cf9c7b4dd595d5e1ef9f256e74 in term 0.
I20260620 20:14:02.206815 32701 raft_consensus.cc:504] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.207145 32701 leader_election.cc:288] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [CANDIDATE]: Term 1 election: Requested vote from peers fa542aa98fb544329ed5ed189e416d0e (127.24.20.129:44977), f7fadf15df8942e9808a38718ebaa937 (127.24.20.131:33427)
I20260620 20:14:02.207459 32339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7373d4fb2dc4d4ca373dffba82b7a82" candidate_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa542aa98fb544329ed5ed189e416d0e"
I20260620 20:14:02.207507 32619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7373d4fb2dc4d4ca373dffba82b7a82" candidate_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fadf15df8942e9808a38718ebaa937"
I20260620 20:14:02.207580 32339 raft_consensus.cc:3037] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:02.207602 32619 raft_consensus.cc:3037] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:02.208665 32619 raft_consensus.cc:2454] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d16d8cf9c7b4dd595d5e1ef9f256e74 in term 1.
I20260620 20:14:02.208778 32339 raft_consensus.cc:2454] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d16d8cf9c7b4dd595d5e1ef9f256e74 in term 1.
I20260620 20:14:02.208954 32412 leader_election.cc:302] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d16d8cf9c7b4dd595d5e1ef9f256e74, f7fadf15df8942e9808a38718ebaa937; no voters:
I20260620 20:14:02.209120 32701 raft_consensus.cc:2781] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:02.209308 32701 raft_consensus.cc:686] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 LEADER]: Becoming Leader. State: Replica: 9d16d8cf9c7b4dd595d5e1ef9f256e74, State: Running, Role: LEADER
I20260620 20:14:02.209583 32701 consensus_queue.cc:237] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.211915 32181 catalog_manager.cc:5495] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130). New cstate: current_term: 1 leader_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } health_report { overall_health: UNKNOWN } } }
I20260620 20:14:02.253825 313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.250202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.24.20.130:38935)
0620 20:14:02.250498 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:02.250505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:02.250518 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:14:02.250593 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:02.250595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:02.250669 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:02.250784 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:02.251002 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.251564 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.252392 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.252674 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.252751 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.252978 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:02.253429 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:02.253514 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:02.253588 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 20:14:02.253632 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":138,"threads_started":1}
I20260620 20:14:02.256493 32619 raft_consensus.cc:1261] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Refusing update from remote peer 9d16d8cf9c7b4dd595d5e1ef9f256e74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:02.256796 32339 raft_consensus.cc:1261] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Refusing update from remote peer 9d16d8cf9c7b4dd595d5e1ef9f256e74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:02.257061 310 consensus_queue.cc:1035] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:02.257323 32701 consensus_queue.cc:1035] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:02.267226 32182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:02.267378 32182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:02.269342 32182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995: Sending DeleteTablet for 3 replicas of tablet b7373d4fb2dc4d4ca373dffba82b7a82
I20260620 20:14:02.269821 32319 tablet_service.cc:1514] Processing DeleteTablet for tablet b7373d4fb2dc4d4ca373dffba82b7a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:02 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:47428
I20260620 20:14:02.269968 32459 tablet_service.cc:1514] Processing DeleteTablet for tablet b7373d4fb2dc4d4ca373dffba82b7a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:02 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:50010
I20260620 20:14:02.270020 32753 tablet_replica.cc:330] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e: stopping tablet replica
I20260620 20:14:02.270148 32753 raft_consensus.cc:2229] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:02.270207 32754 tablet_replica.cc:330] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: stopping tablet replica
I20260620 20:14:02.270200 32599 tablet_service.cc:1514] Processing DeleteTablet for tablet b7373d4fb2dc4d4ca373dffba82b7a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:02 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:45258
I20260620 20:14:02.270270 32753 raft_consensus.cc:2258] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:02.270323 32754 raft_consensus.cc:2229] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:14:02.270360 32755 tablet_replica.cc:330] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937: stopping tablet replica
I20260620 20:14:02.270517 32755 raft_consensus.cc:2229] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:02.270586 32754 raft_consensus.cc:2258] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:02.270651 32755 raft_consensus.cc:2258] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:02.271066 32753 ts_tablet_manager.cc:1905] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:02.271229 32754 ts_tablet_manager.cc:1905] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:02.273154 32753 ts_tablet_manager.cc:1918] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:02.273237 32753 log.cc:1196] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-0/wal/wals/b7373d4fb2dc4d4ca373dffba82b7a82
I20260620 20:14:02.273490 32753 ts_tablet_manager.cc:1939] T b7373d4fb2dc4d4ca373dffba82b7a82 P fa542aa98fb544329ed5ed189e416d0e: Deleting consensus metadata
I20260620 20:14:02.273696 32754 ts_tablet_manager.cc:1918] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:02.273779 32754 log.cc:1196] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/b7373d4fb2dc4d4ca373dffba82b7a82
I20260620 20:14:02.274042 32754 ts_tablet_manager.cc:1939] T b7373d4fb2dc4d4ca373dffba82b7a82 P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Deleting consensus metadata
I20260620 20:14:02.274151 32168 catalog_manager.cc:4841] TS fa542aa98fb544329ed5ed189e416d0e (127.24.20.129:44977): tablet b7373d4fb2dc4d4ca373dffba82b7a82 (table test-table [id=ca7870b571a54b288aee3c240c2c310d]) successfully deleted
I20260620 20:14:02.274669 32166 catalog_manager.cc:4841] TS 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130:38935): tablet b7373d4fb2dc4d4ca373dffba82b7a82 (table test-table [id=ca7870b571a54b288aee3c240c2c310d]) successfully deleted
I20260620 20:14:02.274726 32755 ts_tablet_manager.cc:1905] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:02.275712 32182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:14:02.276229 32182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:14:02.276742 32755 ts_tablet_manager.cc:1918] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:02.276804 32755 log.cc:1196] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/b7373d4fb2dc4d4ca373dffba82b7a82
I20260620 20:14:02.277036 32755 ts_tablet_manager.cc:1939] T b7373d4fb2dc4d4ca373dffba82b7a82 P f7fadf15df8942e9808a38718ebaa937: Deleting consensus metadata
I20260620 20:14:02.277558 32168 catalog_manager.cc:4841] TS f7fadf15df8942e9808a38718ebaa937 (127.24.20.131:33427): tablet b7373d4fb2dc4d4ca373dffba82b7a82 (table test-table [id=ca7870b571a54b288aee3c240c2c310d]) successfully deleted
I20260620 20:14:02.280522 32319 tablet_service.cc:1467] Processing CreateTablet for tablet 38ae6682d5204e00a328266eadd5006a (DEFAULT_TABLE table=test-table [id=91620f67b14b4627998a51a94e20fb60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:02.280524 32459 tablet_service.cc:1467] Processing CreateTablet for tablet 38ae6682d5204e00a328266eadd5006a (DEFAULT_TABLE table=test-table [id=91620f67b14b4627998a51a94e20fb60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:02.280658 32599 tablet_service.cc:1467] Processing CreateTablet for tablet 38ae6682d5204e00a328266eadd5006a (DEFAULT_TABLE table=test-table [id=91620f67b14b4627998a51a94e20fb60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:02.280776 32319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ae6682d5204e00a328266eadd5006a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:02.280776 32459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ae6682d5204e00a328266eadd5006a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:02.280880 32599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ae6682d5204e00a328266eadd5006a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:02.282871 305 tablet_bootstrap.cc:492] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Bootstrap starting.
I20260620 20:14:02.282930 303 tablet_bootstrap.cc:492] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937: Bootstrap starting.
I20260620 20:14:02.283612 305 tablet_bootstrap.cc:654] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:02.283763 303 tablet_bootstrap.cc:654] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:02.284111 301 tablet_bootstrap.cc:492] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e: Bootstrap starting.
I20260620 20:14:02.285180 301 tablet_bootstrap.cc:654] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:02.285912 305 tablet_bootstrap.cc:492] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74: No bootstrap required, opened a new log
I20260620 20:14:02.286006 305 ts_tablet_manager.cc:1397] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 20:14:02.286149 301 tablet_bootstrap.cc:492] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e: No bootstrap required, opened a new log
I20260620 20:14:02.286230 301 ts_tablet_manager.cc:1397] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:14:02.286427 305 raft_consensus.cc:348] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.286543 305 raft_consensus.cc:374] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:02.286582 305 raft_consensus.cc:729] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d16d8cf9c7b4dd595d5e1ef9f256e74, State: Initialized, Role: FOLLOWER
I20260620 20:14:02.286576 301 raft_consensus.cc:348] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.286670 301 raft_consensus.cc:374] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:02.286710 301 raft_consensus.cc:729] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa542aa98fb544329ed5ed189e416d0e, State: Initialized, Role: FOLLOWER
I20260620 20:14:02.286693 305 consensus_queue.cc:260] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.286825 301 consensus_queue.cc:260] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.286975 305 ts_tablet_manager.cc:1428] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 20:14:02.287129 301 ts_tablet_manager.cc:1428] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:14:02.288426 303 tablet_bootstrap.cc:492] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937: No bootstrap required, opened a new log
I20260620 20:14:02.288534 303 ts_tablet_manager.cc:1397] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 20:14:02.288919 303 raft_consensus.cc:348] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.289024 303 raft_consensus.cc:374] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:02.289063 303 raft_consensus.cc:729] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fadf15df8942e9808a38718ebaa937, State: Initialized, Role: FOLLOWER
I20260620 20:14:02.289191 303 consensus_queue.cc:260] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.289433 303 ts_tablet_manager.cc:1428] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:14:02.326884 32699 raft_consensus.cc:482] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:14:02.327001 32699 raft_consensus.cc:504] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.328092 32699 leader_election.cc:288] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa542aa98fb544329ed5ed189e416d0e (127.24.20.129:44977), 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130:38935)
I20260620 20:14:02.332543 32742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.328552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:38365 (local address 127.24.20.129:44977)
0620 20:14:02.328701 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:02.328705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:02.328843 (+ 138us) server_negotiation.cc:407] Connection header received
0620 20:14:02.329022 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:02.329025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:02.329075 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:02.329120 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:02.329538 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.330034 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.331441 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.331970 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.332116 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.332172 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:02.332352 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 20:14:02.332423 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:14:02.332559 320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.328282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:44977 (local address 127.24.20.131:38365)
0620 20:14:02.328745 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:02.328760 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:02.328874 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:02.329282 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:02.329284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:02.329295 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:02.329387 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:02.329393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.330179 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.330182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:02.331296 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:02.331306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.332135 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.332138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:02.332232 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.332257 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:14:02.332327 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:14:02.332393 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":165,"thread_start_us":79,"threads_started":1}
I20260620 20:14:02.333158 32339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae6682d5204e00a328266eadd5006a" candidate_uuid: "f7fadf15df8942e9808a38718ebaa937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa542aa98fb544329ed5ed189e416d0e" is_pre_election: true
I20260620 20:14:02.333326 32339 raft_consensus.cc:2454] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7fadf15df8942e9808a38718ebaa937 in term 0.
I20260620 20:14:02.333504 321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.328321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:38935 (local address 127.24.20.131:34445)
0620 20:14:02.328842 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:02.328853 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:02.328934 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:02.329190 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:02.329193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:02.329226 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:02.329365 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:02.329375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.330460 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.330463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:02.331955 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:02.331966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.333087 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.333091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:02.333200 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.333222 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:14:02.333302 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 20:14:02.333371 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":5,"thread_start_us":100,"threads_started":1}
I20260620 20:14:02.333603 313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.328531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:34445 (local address 127.24.20.130:38935)
0620 20:14:02.328705 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:02.328710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:02.328923 (+ 213us) server_negotiation.cc:407] Connection header received
0620 20:14:02.329027 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:02.329030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:02.329080 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:02.329167 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:02.329481 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.330318 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.332084 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.332693 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.332820 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.332874 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:02.333334 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 20:14:02.333429 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 20:14:02.333853 32552 leader_election.cc:302] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f7fadf15df8942e9808a38718ebaa937, fa542aa98fb544329ed5ed189e416d0e; no voters:
I20260620 20:14:02.334132 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae6682d5204e00a328266eadd5006a" candidate_uuid: "f7fadf15df8942e9808a38718ebaa937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" is_pre_election: true
I20260620 20:14:02.334290 32478 raft_consensus.cc:2454] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7fadf15df8942e9808a38718ebaa937 in term 0.
I20260620 20:14:02.334517 32699 raft_consensus.cc:2781] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:14:02.334585 32699 raft_consensus.cc:482] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:14:02.334633 32699 raft_consensus.cc:3037] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:02.335628 32699 raft_consensus.cc:504] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.335973 32699 leader_election.cc:288] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [CANDIDATE]: Term 1 election: Requested vote from peers fa542aa98fb544329ed5ed189e416d0e (127.24.20.129:44977), 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130:38935)
I20260620 20:14:02.336313 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae6682d5204e00a328266eadd5006a" candidate_uuid: "f7fadf15df8942e9808a38718ebaa937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74"
I20260620 20:14:02.336319 32339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae6682d5204e00a328266eadd5006a" candidate_uuid: "f7fadf15df8942e9808a38718ebaa937" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa542aa98fb544329ed5ed189e416d0e"
I20260620 20:14:02.336434 32478 raft_consensus.cc:3037] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:02.336464 32339 raft_consensus.cc:3037] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:02.337380 32339 raft_consensus.cc:2454] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7fadf15df8942e9808a38718ebaa937 in term 1.
I20260620 20:14:02.337601 32478 raft_consensus.cc:2454] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7fadf15df8942e9808a38718ebaa937 in term 1.
I20260620 20:14:02.337651 32552 leader_election.cc:302] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f7fadf15df8942e9808a38718ebaa937, fa542aa98fb544329ed5ed189e416d0e; no voters:
I20260620 20:14:02.337808 32699 raft_consensus.cc:2781] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:02.338027 32699 raft_consensus.cc:686] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [term 1 LEADER]: Becoming Leader. State: Replica: f7fadf15df8942e9808a38718ebaa937, State: Running, Role: LEADER
I20260620 20:14:02.338269 32699 consensus_queue.cc:237] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } }
I20260620 20:14:02.340440 32182 catalog_manager.cc:5495] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7fadf15df8942e9808a38718ebaa937 (127.24.20.131). New cstate: current_term: 1 leader_uuid: "f7fadf15df8942e9808a38718ebaa937" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7fadf15df8942e9808a38718ebaa937" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33427 } health_report { overall_health: HEALTHY } } }
I20260620 20:14:02.361248 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.357181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.24.20.131:33427)
0620 20:14:02.357551 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:02.357556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:02.357567 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:14:02.357639 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:02.357642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:02.357699 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:02.357805 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:02.358064 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.358829 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.359652 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.359959 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.360069 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.360281 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:02.360874 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:02.360990 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:02.361049 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 20:14:02.361103 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301}
I20260620 20:14:02.372422 32742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.368710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:52261 (local address 127.24.20.129:44977)
0620 20:14:02.368846 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:02.368851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:02.368865 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:14:02.368912 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:02.368915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:02.368964 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:02.369053 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:02.369302 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.370052 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.371232 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.371580 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.371680 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.371971 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:02.372064 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:02.372143 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:02.372235 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 20:14:02.372288 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:14:02.372809 326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.368217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:44977 (local address 127.24.20.131:52261)
0620 20:14:02.368517 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:02.368532 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:02.368654 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:02.369067 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:02.369070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:02.369078 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:14:02.369188 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:02.369194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.370163 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.370166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:02.371103 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:02.371112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.371690 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.371693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:02.371786 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.371831 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:14:02.372157 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:14:02.372162 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:14:02.372222 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 20:14:02.372294 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":109,"threads_started":1}
I20260620 20:14:02.379081 32478 raft_consensus.cc:1261] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Refusing update from remote peer f7fadf15df8942e9808a38718ebaa937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:02.379423 32339 raft_consensus.cc:1261] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Refusing update from remote peer f7fadf15df8942e9808a38718ebaa937: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:02.379611 32699 consensus_queue.cc:1035] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d16d8cf9c7b4dd595d5e1ef9f256e74" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:02.379859 322 consensus_queue.cc:1035] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa542aa98fb544329ed5ed189e416d0e" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 44977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:02.398730 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.395318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:40311 (local address 127.24.20.131:33427)
0620 20:14:02.395421 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:02.395425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:02.395483 (+ 58us) server_negotiation.cc:407] Connection header received
0620 20:14:02.395635 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:02.395639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:02.395687 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:02.395765 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:02.396112 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.396801 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.397734 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.397977 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.398082 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.398303 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:02.398377 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:02.398452 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:02.398552 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 20:14:02.398598 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:14:02.398741 32750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.395248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:33427 (local address 127.24.20.129:40311)
0620 20:14:02.395421 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:02.395436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:02.395528 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:02.395873 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:02.395876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:02.395884 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:14:02.395978 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:02.395985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.396907 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.396910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:02.397617 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:02.397626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:02.398079 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:02.398083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:02.398168 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:02.398197 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:14:02.398454 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:14:02.398463 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 20:14:02.398527 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 20:14:02.398605 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 20:14:02.418269 32180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:02.418437 32180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:02.420763 32180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e742953c59a74113ae92276324d9b995: Sending DeleteTablet for 3 replicas of tablet 38ae6682d5204e00a328266eadd5006a
I20260620 20:14:02.421388 32599 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ae6682d5204e00a328266eadd5006a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:02 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:45258
I20260620 20:14:02.421428 32319 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ae6682d5204e00a328266eadd5006a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:02 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:47428
I20260620 20:14:02.421561 32755 tablet_replica.cc:330] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937: stopping tablet replica
I20260620 20:14:02.421598 32753 tablet_replica.cc:330] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e: stopping tablet replica
I20260620 20:14:02.421700 32755 raft_consensus.cc:2229] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:14:02.421713 32753 raft_consensus.cc:2229] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:02.421811 32753 raft_consensus.cc:2258] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:02.421967 32755 raft_consensus.cc:2258] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:02.422571 32753 ts_tablet_manager.cc:1905] T 38ae6682d5204e00a328266eadd5006a P fa542aa98fb544329ed5ed189e416d0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:02.422927 32755 ts_tablet_manager.cc:1905] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:02.423079 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 32254
I20260620 20:14:02.425449 32755 ts_tablet_manager.cc:1918] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:14:02.425514 32755 log.cc:1196] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/38ae6682d5204e00a328266eadd5006a
I20260620 20:14:02.425765 32755 ts_tablet_manager.cc:1939] T 38ae6682d5204e00a328266eadd5006a P f7fadf15df8942e9808a38718ebaa937: Deleting consensus metadata
I20260620 20:14:02.426420 32168 catalog_manager.cc:4841] TS f7fadf15df8942e9808a38718ebaa937 (127.24.20.131:33427): tablet 38ae6682d5204e00a328266eadd5006a (table test-table [id=91620f67b14b4627998a51a94e20fb60]) successfully deleted
I20260620 20:14:02.426715 32459 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ae6682d5204e00a328266eadd5006a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:02 UTC) from {username='oryx', principal='oryx/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:50010
I20260620 20:14:02.426865 32754 tablet_replica.cc:330] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74: stopping tablet replica
I20260620 20:14:02.426957 32754 raft_consensus.cc:2229] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:02.427062 32754 raft_consensus.cc:2258] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:02.427664 32754 ts_tablet_manager.cc:1905] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:02.429878 32754 ts_tablet_manager.cc:1918] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:14:02.429953 32754 log.cc:1196] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/38ae6682d5204e00a328266eadd5006a
I20260620 20:14:02.430193 32754 ts_tablet_manager.cc:1939] T 38ae6682d5204e00a328266eadd5006a P 9d16d8cf9c7b4dd595d5e1ef9f256e74: Deleting consensus metadata
I20260620 20:14:02.430853 32166 catalog_manager.cc:4841] TS 9d16d8cf9c7b4dd595d5e1ef9f256e74 (127.24.20.130:38935): tablet 38ae6682d5204e00a328266eadd5006a (table test-table [id=91620f67b14b4627998a51a94e20fb60]) successfully deleted
W20260620 20:14:02.435750 32168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:44977 (error 108)
I20260620 20:14:02.436044 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 32394
I20260620 20:14:02.437216 334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:02.436738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:44977 (local address 127.0.0.1:47458)
0620 20:14:02.437058 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:02.437122 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:44977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":48,"threads_started":1}
W20260620 20:14:02.437310 32168 catalog_manager.cc:4584] TS fa542aa98fb544329ed5ed189e416d0e (127.24.20.129:44977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38ae6682d5204e00a328266eadd5006a: Network error: Client connection negotiation failed: client connection to 127.24.20.129:44977: connect: Connection refused (error 111)
I20260620 20:14:02.441771 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 32534
I20260620 20:14:02.446745 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 32150
2026-06-20T20:14:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3413 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:02 dist-test-slave-pflr krb5kdc[337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:02 dist-test-slave-pflr krb5kdc[337](info): set up 2 sockets
Jun 20 20:14:02 dist-test-slave-pflr krb5kdc[337](info): commencing operation
krb5kdc: starting...
W20260620 20:14:03.493717 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:14:03 dist-test-slave-pflr krb5kdc[337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:14:03.611151 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:41063
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:41063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:03.720332 350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:03.720636 350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:03.720719 350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:03.724668 350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:03.724736 350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:03.724774 350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:03.724812 350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:03.724849 350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:03.730077 350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:41063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:41063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:03.731056 350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:03.732127 350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:03.740253 358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:03.740289 356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:03.740471 350 server_base.cc:1034] running on GCE node
W20260620 20:14:03.740269 355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:03.741142 350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:03.742202 350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:03.743382 350 hybrid_clock.cc:648] HybridClock initialized: now 1781986443743357 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T20:14:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1301 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:03 dist-test-slave-pflr krb5kdc[363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:03 dist-test-slave-pflr krb5kdc[363](info): set up 2 sockets
Jun 20 20:14:03 dist-test-slave-pflr krb5kdc[363](info): commencing operation
krb5kdc: starting...
W20260620 20:14:04.789608 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:14:04 dist-test-slave-pflr krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:14:04.911912 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:42177
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:05.017932 376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:05.018183 376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:05.018222 376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:05.022076 376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:05.022125 376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:05.022145 376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:05.022161 376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:05.022176 376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:05.027173 376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:42177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:42177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:05.028069 376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:05.029135 376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:05.037066 381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:05.037060 384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:05.037081 382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:05.037101 376 server_base.cc:1034] running on GCE node
I20260620 20:14:05.037808 376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:05.038820 376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:05.039991 376 hybrid_clock.cc:648] HybridClock initialized: now 1781986445039957 us; error 45 us; skew 500 ppm
Jun 20 20:14:05 dist-test-slave-pflr krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986445, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:05.042871 376 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:14:05.043715 376 webserver.cc:460] Webserver started at http://127.24.20.190:46433/ using document root <none> and password file <none>
I20260620 20:14:05.044256 376 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:05.044317 376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:05.044521 376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:05.046543 376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "d35f0c17abef44de9c275d8a404d50dc"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "a4da28e4f9bfd346f5cd6b5dd69217f5"
server_key_iv: "1fb99abfb2e8d27a0c1ad7ed28d6cd71"
server_key_version: "encryptionkey@0"
I20260620 20:14:05.047128 376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "d35f0c17abef44de9c275d8a404d50dc"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "a4da28e4f9bfd346f5cd6b5dd69217f5"
server_key_iv: "1fb99abfb2e8d27a0c1ad7ed28d6cd71"
server_key_version: "encryptionkey@0"
I20260620 20:14:05.050678 376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:14:05.052963 391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:05.054138 376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:14:05.054263 376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "d35f0c17abef44de9c275d8a404d50dc"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "a4da28e4f9bfd346f5cd6b5dd69217f5"
server_key_iv: "1fb99abfb2e8d27a0c1ad7ed28d6cd71"
server_key_version: "encryptionkey@0"
I20260620 20:14:05.054366 376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:05.065567 376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:05.068492 376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:05.068717 376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:05.075047 376 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:42177
I20260620 20:14:05.075120 443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:42177 every 8 connection(s)
I20260620 20:14:05.075963 376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:05.077317 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 376
I20260620 20:14:05.077461 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:14:05.077745 24658 external_mini_cluster.cc:1428] Setting key 8ef002ced395f96cdfe74177fcb83ddf
I20260620 20:14:05.078977 444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:05.086246 444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc: Bootstrap starting.
I20260620 20:14:05.088922 444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:05.089862 444 log.cc:824] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:14:05 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:14:05.091676 444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc: No bootstrap required, opened a new log
I20260620 20:14:05.095090 444 raft_consensus.cc:348] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f0c17abef44de9c275d8a404d50dc" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42177 } }
I20260620 20:14:05.095295 444 raft_consensus.cc:374] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:05.095351 444 raft_consensus.cc:729] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35f0c17abef44de9c275d8a404d50dc, State: Initialized, Role: FOLLOWER
I20260620 20:14:05.095386 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:05.079517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.24.20.190:42177)
0620 20:14:05.080038 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:05.080082 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:05.082326 (+ 2244us) server_negotiation.cc:407] Connection header received
0620 20:14:05.083274 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:05.083297 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:05.083701 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:05.084057 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:05.085418 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.086416 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.087617 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.088063 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.088178 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:05.090995 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:05.091000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:05.091012 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:05.091037 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:05.092746 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:05.093170 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:05.093176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:05.093180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:05.093282 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:05.093556 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:05.093560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:05.093561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:05.093916 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:05.094129 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:05.094360 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 20:14:05.094654 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":159,"threads_started":1}
I20260620 20:14:05.095868 444 consensus_queue.cc:260] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f0c17abef44de9c275d8a404d50dc" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42177 } }
I20260620 20:14:05.096029 444 raft_consensus.cc:388] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:05.096113 444 raft_consensus.cc:482] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:05.096231 444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:05.097301 444 raft_consensus.cc:504] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f0c17abef44de9c275d8a404d50dc" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42177 } }
I20260620 20:14:05.097647 444 leader_election.cc:302] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d35f0c17abef44de9c275d8a404d50dc; no voters:
I20260620 20:14:05.097945 444 leader_election.cc:288] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:05.098038 449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:05.098261 449 raft_consensus.cc:686] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 1 LEADER]: Becoming Leader. State: Replica: d35f0c17abef44de9c275d8a404d50dc, State: Running, Role: LEADER
I20260620 20:14:05.098604 449 consensus_queue.cc:237] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f0c17abef44de9c275d8a404d50dc" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42177 } }
I20260620 20:14:05.099162 444 sys_catalog.cc:564] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:05.100155 450 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d35f0c17abef44de9c275d8a404d50dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f0c17abef44de9c275d8a404d50dc" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42177 } } }
I20260620 20:14:05.100278 450 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:05.100646 451 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d35f0c17abef44de9c275d8a404d50dc. Latest consensus state: current_term: 1 leader_uuid: "d35f0c17abef44de9c275d8a404d50dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f0c17abef44de9c275d8a404d50dc" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42177 } } }
I20260620 20:14:05.100683 458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:05.100795 451 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:05.103804 458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:05.108680 458 catalog_manager.cc:1347] Generated new cluster ID: 20da8bee369746c39c0741f69520d245
I20260620 20:14:05.108753 458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:05.117909 458 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:05.118824 458 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:05.131283 458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc: Generated new TSK 0
I20260620 20:14:05.132025 458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:14:05.169646 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42177
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:05.280838 470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:05.281080 470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:05.281122 470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:05.285130 470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:05.285185 470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:05.285243 470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:14:05.290354 470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:05.291275 470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:05.292201 470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:05.301640 475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:05.301640 476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:05.301633 478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:05.301843 470 server_base.cc:1034] running on GCE node
I20260620 20:14:05.302284 470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:05.302927 470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:05.304104 470 hybrid_clock.cc:648] HybridClock initialized: now 1781986445304088 us; error 27 us; skew 500 ppm
Jun 20 20:14:05 dist-test-slave-pflr krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986445, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:05.306756 470 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:14:05.307526 470 webserver.cc:460] Webserver started at http://127.24.20.129:35291/ using document root <none> and password file <none>
I20260620 20:14:05.308017 470 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:05.308080 470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:05.308233 470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:05.310225 470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "e68e365d82d34e7391e840705aea80ab"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "9c7396a199da38d980d145f6798baa95"
server_key_iv: "63349068b4e5b478dc87e51accd780c6"
server_key_version: "encryptionkey@0"
I20260620 20:14:05.310716 470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "e68e365d82d34e7391e840705aea80ab"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "9c7396a199da38d980d145f6798baa95"
server_key_iv: "63349068b4e5b478dc87e51accd780c6"
server_key_version: "encryptionkey@0"
I20260620 20:14:05.314220 470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:14:05.316370 485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:05.317569 470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:14:05.317684 470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "e68e365d82d34e7391e840705aea80ab"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "9c7396a199da38d980d145f6798baa95"
server_key_iv: "63349068b4e5b478dc87e51accd780c6"
server_key_version: "encryptionkey@0"
I20260620 20:14:05.317780 470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:05.330459 470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:05.337102 470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:05.337280 470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:05.338094 470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:05.338881 470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:05.338928 470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:05.339015 470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:05.339046 470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:05.347823 470 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:46435
I20260620 20:14:05.347848 598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:46435 every 8 connection(s)
I20260620 20:14:05.348701 470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:14:05.355670 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 470
I20260620 20:14:05.355810 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:14:05.356115 24658 external_mini_cluster.cc:1428] Setting key b659bc8bb3f012f3aafb6fdc53a180bf
Jun 20 20:14:05 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986445, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:14:05.362138 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:05.350314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:50873 (local address 127.24.20.190:42177)
0620 20:14:05.350432 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:05.350437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:05.351159 (+ 722us) server_negotiation.cc:407] Connection header received
0620 20:14:05.352473 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:05.352476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:05.352535 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:05.352633 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:05.353751 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.354291 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.355404 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.355654 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.355758 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:05.358374 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:05.358381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:05.358387 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:05.358421 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:05.359892 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:05.360493 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:05.360496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:05.360497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:05.360666 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:05.361034 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:05.361036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:05.361037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:05.361194 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:05.361268 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:05.361916 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 20:14:05.362022 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:14:05.363359 601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:05.350619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:50873)
0620 20:14:05.351021 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:05.351060 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:05.352264 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:05.352787 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:05.352793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:05.353121 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:05.353610 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:05.353623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.354412 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.354416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:05.355275 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:05.355284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.355806 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.355810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:05.355999 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:05.356554 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:05.356570 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:05.358226 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:05.360001 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:05.360008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:05.360022 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:05.360400 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:05.360754 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:05.360761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:05.360766 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:05.360945 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:05.361291 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:05.361300 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:05.361648 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 20:14:05.361858 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 20:14:05.362218 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":105,"threads_started":1}
I20260620 20:14:05.364666 599 heartbeater.cc:347] Connected to a master server at 127.24.20.190:42177
I20260620 20:14:05.364888 599 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:05.365345 599 heartbeater.cc:511] Master 127.24.20.190:42177 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:14:05.366817 408 ts_manager.cc:194] Registered new tserver with Master: e68e365d82d34e7391e840705aea80ab (127.24.20.129:46435)
I20260620 20:14:05.367923 408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:50873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:14:05.383294 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42177
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:05.488624 604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:05.488849 604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:05.488889 604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:05.492724 604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:05.492774 604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:05.492826 604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:14:05.498373 604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:05.499354 604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:05.500371 604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:05.509506 610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:05.509547 609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:05.509639 612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:05.509956 604 server_base.cc:1034] running on GCE node
I20260620 20:14:05.510362 604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:05.510979 604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:05.512178 604 hybrid_clock.cc:648] HybridClock initialized: now 1781986445512170 us; error 29 us; skew 500 ppm
Jun 20 20:14:05 dist-test-slave-pflr krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986445, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:05.515125 604 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:14:05.516064 604 webserver.cc:460] Webserver started at http://127.24.20.130:32971/ using document root <none> and password file <none>
I20260620 20:14:05.516687 604 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:05.516776 604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:05.516997 604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:05.519079 604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "651a6fba515d4c1fa9a289fc8212a72a"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "15e790c7befea25144182bc801a85cef"
server_key_iv: "49d73c567fcf766c84d58c8acf3e06d9"
server_key_version: "encryptionkey@0"
I20260620 20:14:05.519639 604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "651a6fba515d4c1fa9a289fc8212a72a"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "15e790c7befea25144182bc801a85cef"
server_key_iv: "49d73c567fcf766c84d58c8acf3e06d9"
server_key_version: "encryptionkey@0"
I20260620 20:14:05.523427 604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 20:14:05.525882 619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:05.527076 604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:14:05.527202 604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "651a6fba515d4c1fa9a289fc8212a72a"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "15e790c7befea25144182bc801a85cef"
server_key_iv: "49d73c567fcf766c84d58c8acf3e06d9"
server_key_version: "encryptionkey@0"
I20260620 20:14:05.527302 604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:05.547943 604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:05.551419 604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:05.551621 604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:05.552186 604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:05.553007 604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:05.553076 604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:05.553140 604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:05.553186 604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:05.562274 604 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:46489
I20260620 20:14:05.562295 732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:46489 every 8 connection(s)
I20260620 20:14:05.563134 604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:14:05.568945 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 604
I20260620 20:14:05.569073 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:14:05.569420 24658 external_mini_cluster.cc:1428] Setting key 3fcdbaed94d4887b6e3201e22b8276c5
Jun 20 20:14:05 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986445, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:14:05.576876 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:05.564937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:38381 (local address 127.24.20.190:42177)
0620 20:14:05.565059 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:05.565063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:05.565850 (+ 787us) server_negotiation.cc:407] Connection header received
0620 20:14:05.567191 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:05.567195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:05.567246 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:05.567329 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:05.568421 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.569232 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.570497 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.570819 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.570914 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:05.573315 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:05.573318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:05.573320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:05.573340 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:05.574823 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:05.575418 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:05.575422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:05.575424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:05.575509 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:05.575823 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:05.575828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:05.575829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:05.576019 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:05.576121 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:05.576585 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 20:14:05.576718 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:14:05.577689 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:05.565298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:38381)
0620 20:14:05.565719 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:05.565758 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:05.566995 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:05.567493 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:05.567499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:05.567834 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:05.568285 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:05.568297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.569354 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.569358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:05.570381 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:05.570391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.570965 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.570968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:05.571119 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:05.571717 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:05.571732 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:05.573153 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:05.574951 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:05.574959 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:05.574974 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:05.575320 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:05.575602 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:05.575605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:05.575607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:05.575734 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:05.576117 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:05.576123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:05.576345 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 20:14:05.576539 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 20:14:05.576869 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
I20260620 20:14:05.578716 733 heartbeater.cc:347] Connected to a master server at 127.24.20.190:42177
I20260620 20:14:05.578938 733 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:05.579408 733 heartbeater.cc:511] Master 127.24.20.190:42177 requested a full tablet report, sending...
I20260620 20:14:05.580458 408 ts_manager.cc:194] Registered new tserver with Master: 651a6fba515d4c1fa9a289fc8212a72a (127.24.20.130:46489)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
I20260620 20:14:05.581152 408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:38381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:14:05.596130 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42177
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:05.705233 738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:05.705459 738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:05.705499 738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:05.709414 738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:05.709465 738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:05.709520 738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:14:05.714546 738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:05.715426 738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:05.716334 738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:05.725355 746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:05.725338 744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:05.725391 743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:05.725699 738 server_base.cc:1034] running on GCE node
I20260620 20:14:05.726128 738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:05.726744 738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:05.727905 738 hybrid_clock.cc:648] HybridClock initialized: now 1781986445727879 us; error 36 us; skew 500 ppm
Jun 20 20:14:05 dist-test-slave-pflr krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986445, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:05.730801 738 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:14:05.731675 738 webserver.cc:460] Webserver started at http://127.24.20.131:46389/ using document root <none> and password file <none>
I20260620 20:14:05.732263 738 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:05.732323 738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:05.732482 738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:05.734421 738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "d43d39368b0742b69b3e69dae35d073e"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "66c7f5936c1466ca4a33fd0505a22a06"
server_key_iv: "811483ccc8880e8029cf3d9afc14d929"
server_key_version: "encryptionkey@0"
I20260620 20:14:05.734921 738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "d43d39368b0742b69b3e69dae35d073e"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "66c7f5936c1466ca4a33fd0505a22a06"
server_key_iv: "811483ccc8880e8029cf3d9afc14d929"
server_key_version: "encryptionkey@0"
I20260620 20:14:05.738425 738 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:14:05.740768 753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:05.742017 738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:14:05.742142 738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "d43d39368b0742b69b3e69dae35d073e"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "66c7f5936c1466ca4a33fd0505a22a06"
server_key_iv: "811483ccc8880e8029cf3d9afc14d929"
server_key_version: "encryptionkey@0"
I20260620 20:14:05.742247 738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:05.763320 738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:05.766634 738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:05.766827 738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:05.767388 738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:05.768203 738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:05.768276 738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:05.768345 738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:05.768393 738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:05.777287 738 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:42585
I20260620 20:14:05.777341 866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:42585 every 8 connection(s)
I20260620 20:14:05.778211 738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:14:05.781848 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 738
I20260620 20:14:05.782007 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:14:05.782307 24658 external_mini_cluster.cc:1428] Setting key 4ceddfb9463e4ce06019d72f2f88002c
Jun 20 20:14:05 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986445, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:14:05.792927 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:05.780108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:47233 (local address 127.24.20.190:42177)
0620 20:14:05.780254 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:05.780258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:05.781180 (+ 922us) server_negotiation.cc:407] Connection header received
0620 20:14:05.782823 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:05.782826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:05.782900 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:05.782996 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:05.784689 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.785252 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.786436 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.786673 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.786759 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:05.789470 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:05.789474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:05.789476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:05.789500 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:05.790850 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:05.791342 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:05.791344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:05.791346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:05.791429 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:05.791822 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:05.791825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:05.791826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:05.792004 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:05.792085 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:05.792695 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 20:14:05.792798 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:14:05.793879 869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:05.780492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:47233)
0620 20:14:05.781011 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:05.781055 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:05.782573 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:05.783214 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:05.783225 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:05.783825 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:05.784522 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:05.784545 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.785383 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.785386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:05.786310 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:05.786320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:05.786807 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:05.786809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:05.786955 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:05.787552 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:05.787569 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:05.789290 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:05.790974 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:05.790981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:05.790993 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:05.791246 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:05.791554 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:05.791557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:05.791558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:05.791694 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:05.792120 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:05.792126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:05.792367 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 20:14:05.792577 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 20:14:05.792925 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":124,"threads_started":1}
I20260620 20:14:05.795149 867 heartbeater.cc:347] Connected to a master server at 127.24.20.190:42177
I20260620 20:14:05.795462 867 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:05.796095 867 heartbeater.cc:511] Master 127.24.20.190:42177 requested a full tablet report, sending...
I20260620 20:14:05.797209 408 ts_manager.cc:194] Registered new tserver with Master: d43d39368b0742b69b3e69dae35d073e (127.24.20.131:42585)
I20260620 20:14:05.797796 408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:47233
I20260620 20:14:05.808663 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab.
Entry for principal oryx/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab.
WARNING: no policy specified for oryx/127.24.20.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.20.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab.
Entry for principal oryx/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab.
WARNING: no policy specified for oryx/127.24.20.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.20.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab.
Entry for principal oryx/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab.
I20260620 20:14:05.883396 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 470
I20260620 20:14:05.889640 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 604
I20260620 20:14:05.894943 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 738
I20260620 20:14:05.902235 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 376
I20260620 20:14:05.908061 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:42177
--webserver_interface=127.24.20.190
--webserver_port=46433
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:06.016963 876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:06.017256 876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:06.017325 876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:06.021306 876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:06.021384 876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:06.021421 876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:06.021472 876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:06.021515 876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:06.026605 876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:42177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:42177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=46433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:06.027624 876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:06.028657 876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:06.037158 884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:06.037164 882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:06.037218 881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:06.037307 876 server_base.cc:1034] running on GCE node
I20260620 20:14:06.037946 876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:06.039177 876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:06.040345 876 hybrid_clock.cc:648] HybridClock initialized: now 1781986446040328 us; error 27 us; skew 500 ppm
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986446, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:06.043308 876 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:14:06.044224 876 webserver.cc:460] Webserver started at http://127.24.20.190:46433/ using document root <none> and password file <none>
I20260620 20:14:06.044869 876 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:06.044982 876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:06.048985 876 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:14:06.051184 891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:06.052507 876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:14:06.052717 876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "d35f0c17abef44de9c275d8a404d50dc"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "a4da28e4f9bfd346f5cd6b5dd69217f5"
server_key_iv: "1fb99abfb2e8d27a0c1ad7ed28d6cd71"
server_key_version: "encryptionkey@0"
I20260620 20:14:06.053287 876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:06.078644 876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:06.081754 876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:06.081962 876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:06.089398 876 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:42177
I20260620 20:14:06.089417 943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:42177 every 8 connection(s)
I20260620 20:14:06.090467 876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:06.094077 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 876
I20260620 20:14:06.094578 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:46435
--local_ip_for_outbound_sockets=127.24.20.129
--tserver_master_addrs=127.24.20.190:42177
--webserver_port=35291
--webserver_interface=127.24.20.129
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.20.129
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:14:06.095052 944 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:14:06.097195 944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc: Bootstrap starting.
I20260620 20:14:06.113607 944 log.cc:824] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:06.120194 944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:14:06.120579 944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc: Bootstrap complete.
I20260620 20:14:06.124788 944 raft_consensus.cc:348] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f0c17abef44de9c275d8a404d50dc" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42177 } }
I20260620 20:14:06.125208 944 raft_consensus.cc:729] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35f0c17abef44de9c275d8a404d50dc, State: Initialized, Role: FOLLOWER
I20260620 20:14:06.125727 944 consensus_queue.cc:260] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f0c17abef44de9c275d8a404d50dc" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42177 } }
I20260620 20:14:06.125871 944 raft_consensus.cc:388] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:06.125950 944 raft_consensus.cc:482] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:06.126056 944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:14:06.127892 944 raft_consensus.cc:504] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f0c17abef44de9c275d8a404d50dc" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42177 } }
I20260620 20:14:06.128213 944 leader_election.cc:302] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d35f0c17abef44de9c275d8a404d50dc; no voters:
I20260620 20:14:06.128545 944 leader_election.cc:288] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:14:06.128656 948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:14:06.128927 948 raft_consensus.cc:686] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [term 2 LEADER]: Becoming Leader. State: Replica: d35f0c17abef44de9c275d8a404d50dc, State: Running, Role: LEADER
I20260620 20:14:06.129278 948 consensus_queue.cc:237] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f0c17abef44de9c275d8a404d50dc" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42177 } }
I20260620 20:14:06.129777 944 sys_catalog.cc:564] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:06.130801 949 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d35f0c17abef44de9c275d8a404d50dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f0c17abef44de9c275d8a404d50dc" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42177 } } }
I20260620 20:14:06.130856 950 sys_catalog.cc:455] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d35f0c17abef44de9c275d8a404d50dc. Latest consensus state: current_term: 2 leader_uuid: "d35f0c17abef44de9c275d8a404d50dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d35f0c17abef44de9c275d8a404d50dc" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42177 } } }
I20260620 20:14:06.130983 950 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:06.130930 949 sys_catalog.cc:458] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:06.131371 957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:06.134609 957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:06.135290 957 catalog_manager.cc:1259] Loaded cluster ID: 20da8bee369746c39c0741f69520d245
I20260620 20:14:06.135352 957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:06.137683 957 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:06.138339 957 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d35f0c17abef44de9c275d8a404d50dc: Loaded TSK: 0
I20260620 20:14:06.139572 957 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:14:06.210636 946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:06.210913 946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:06.210984 946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:06.214975 946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:06.215058 946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:06.215166 946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:14:06.220332 946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.129.keytab
--principal=oryx/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:46435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=35291
--tserver_master_addrs=127.24.20.190:42177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:06.221369 946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:06.222337 946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:06.232786 973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:06.232771 970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:06.232928 946 server_base.cc:1034] running on GCE node
W20260620 20:14:06.232766 971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:06.233928 946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:06.234582 946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:06.235771 946 hybrid_clock.cc:648] HybridClock initialized: now 1781986446235745 us; error 47 us; skew 500 ppm
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986446, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:06.238584 946 init.cc:377] Logged in from keytab as oryx/127.24.20.129@KRBTEST.COM (short username oryx)
I20260620 20:14:06.239490 946 webserver.cc:460] Webserver started at http://127.24.20.129:35291/ using document root <none> and password file <none>
I20260620 20:14:06.240048 946 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:06.240123 946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:06.244066 946 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:14:06.246264 980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:06.247509 946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:14:06.247642 946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "e68e365d82d34e7391e840705aea80ab"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "9c7396a199da38d980d145f6798baa95"
server_key_iv: "63349068b4e5b478dc87e51accd780c6"
server_key_version: "encryptionkey@0"
I20260620 20:14:06.248160 946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:06.263281 946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:06.266620 946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:06.266813 946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:06.267364 946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:06.268218 946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:06.268282 946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:06.268347 946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:06.268393 946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:06.277700 946 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:46435
I20260620 20:14:06.277724 1093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:46435 every 8 connection(s)
I20260620 20:14:06.278685 946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:14:06.281313 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 946
I20260620 20:14:06.281852 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:46489
--local_ip_for_outbound_sockets=127.24.20.130
--tserver_master_addrs=127.24.20.190:42177
--webserver_port=32971
--webserver_interface=127.24.20.130
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.20.130
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:06.292001 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:06.280852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:57295)
0620 20:14:06.281333 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:06.281384 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:06.282880 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:06.284065 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:06.284073 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:06.284535 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:06.285050 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.285064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.286290 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.286298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.287160 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.287168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.287890 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.287894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.288081 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.288689 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:06.288705 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:06.291069 (+ 2364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":154,"threads_started":1}
W20260620 20:14:06.292011 1097 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:06.280817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:57295 (local address 127.24.20.190:42177)
0620 20:14:06.281428 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:06.281447 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:06.281495 (+ 48us) server_negotiation.cc:407] Connection header received
0620 20:14:06.283357 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:06.283367 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:06.283661 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:06.283976 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:06.285213 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.286158 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.287281 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.287737 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.287903 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.291263 (+ 3360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:57295: BlockingRecv error: recv got EOF from 127.24.20.129:57295 (error 108)
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":142,"threads_started":1}
W20260620 20:14:06.292667 1094 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:42177 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.20.190:42177: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:06.298717 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:06.293115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:35219)
0620 20:14:06.293237 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:06.293251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:06.293330 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:06.293560 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:06.293563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:06.293726 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:06.293827 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.293833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.294602 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.294606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.295522 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.295532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.295982 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.295986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.296076 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.296542 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:06.296556 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:06.298586 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 20:14:06.299002 1097 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:06.293216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:35219 (local address 127.24.20.190:42177)
0620 20:14:06.293313 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:06.293316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:06.293328 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:14:06.293413 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:06.293415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:06.293462 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:06.293528 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:06.293983 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.294503 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.295660 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.295874 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.295965 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.298728 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:35219: BlockingRecv error: recv got EOF from 127.24.20.129:35219 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:06.304622 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:06.299245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:38319)
0620 20:14:06.299333 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:06.299348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:06.299425 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:06.299720 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:06.299722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:06.299876 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:06.299973 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.299978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.300772 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.300776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.301625 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.301636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.302085 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.302089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.302173 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.302667 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:06.302681 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:06.304490 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 20:14:06.304804 1094 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:14:06.304849 1097 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:06.299309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:38319 (local address 127.24.20.190:42177)
0620 20:14:06.299546 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:06.299549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:06.299558 (+ 9us) server_negotiation.cc:407] Connection header received
0620 20:14:06.299585 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:06.299587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:06.299619 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:06.299675 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:06.300106 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.300671 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.301765 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.301973 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.302067 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.304737 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:38319: BlockingRecv error: recv got EOF from 127.24.20.129:38319 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
W20260620 20:14:06.393492 1098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:06.393720 1098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:06.393760 1098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:06.397683 1098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:06.397733 1098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:06.397787 1098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:14:06.402729 1098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.130.keytab
--principal=oryx/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:46489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=32971
--tserver_master_addrs=127.24.20.190:42177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:06.403611 1098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:06.404525 1098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:06.413833 1104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:06.413858 1106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:06.413883 1098 server_base.cc:1034] running on GCE node
W20260620 20:14:06.413815 1103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:06.414467 1098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:06.415087 1098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:06.416239 1098 hybrid_clock.cc:648] HybridClock initialized: now 1781986446416210 us; error 43 us; skew 500 ppm
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986446, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:06.418927 1098 init.cc:377] Logged in from keytab as oryx/127.24.20.130@KRBTEST.COM (short username oryx)
I20260620 20:14:06.419824 1098 webserver.cc:460] Webserver started at http://127.24.20.130:32971/ using document root <none> and password file <none>
I20260620 20:14:06.420346 1098 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:06.420414 1098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:06.424078 1098 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:14:06.426147 1113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:06.427408 1098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:14:06.427574 1098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "651a6fba515d4c1fa9a289fc8212a72a"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "15e790c7befea25144182bc801a85cef"
server_key_iv: "49d73c567fcf766c84d58c8acf3e06d9"
server_key_version: "encryptionkey@0"
I20260620 20:14:06.428094 1098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:06.458144 1098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:06.461195 1098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:06.461390 1098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:06.461973 1098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:06.462795 1098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:06.462862 1098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:06.462932 1098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:06.462971 1098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:06.471907 1098 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:46489
I20260620 20:14:06.471987 1226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:46489 every 8 connection(s)
I20260620 20:14:06.472944 1098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:14:06.477962 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 1098
I20260620 20:14:06.478411 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:42585
--local_ip_for_outbound_sockets=127.24.20.131
--tserver_master_addrs=127.24.20.190:42177
--webserver_port=46389
--webserver_interface=127.24.20.131
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.20.131
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:14:06.484334 1097 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:06.474706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:48027 (local address 127.24.20.190:42177)
0620 20:14:06.474831 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:06.474835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:06.475657 (+ 822us) server_negotiation.cc:407] Connection header received
0620 20:14:06.477056 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:06.477060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:06.477132 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:06.477238 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:06.478487 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.479107 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.480238 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.480484 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.480596 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.484166 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:48027: BlockingRecv error: recv got EOF from 127.24.20.130:48027 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:14:06.484824 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:06.475072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:48027)
0620 20:14:06.475505 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:06.475546 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:06.476834 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:06.477404 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:06.477457 (+ 53us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:06.477807 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:06.478340 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.478355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.479235 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.479239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.480110 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.480119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.480664 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.480669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.480832 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.481455 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:06.481471 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:06.483970 (+ 2499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
W20260620 20:14:06.485386 1227 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:42177 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.20.190:42177: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:06.490545 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:06.485839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:54367)
0620 20:14:06.485943 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:06.485958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:06.486031 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:06.486274 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:06.486277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:06.486436 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:06.486522 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.486526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.487213 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.487217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.487755 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.487763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.488182 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.488185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.488273 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.488758 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:06.488774 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:06.490421 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 20:14:06.490721 1097 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:06.485948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:54367 (local address 127.24.20.190:42177)
0620 20:14:06.486088 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:06.486094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:06.486109 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:14:06.486148 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:06.486150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:06.486182 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:06.486262 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:06.486628 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.487111 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.487868 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.488079 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.488182 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.490606 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:54367: BlockingRecv error: recv got EOF from 127.24.20.130:54367 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:06.496397 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:06.491430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:58261)
0620 20:14:06.491538 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:06.491552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:06.491616 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:06.491828 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:06.491829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:06.491955 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:06.492021 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.492025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.493015 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.493018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.493542 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.493555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.494128 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.494131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.494190 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.494631 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:06.494641 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:06.496313 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 20:14:06.496553 1097 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:06.491495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:58261 (local address 127.24.20.190:42177)
0620 20:14:06.491599 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:06.491603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:06.491617 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:14:06.491676 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:06.491679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:06.491720 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:06.491797 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:06.492113 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.492891 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.493650 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.493985 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.494084 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.496447 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:58261: BlockingRecv error: recv got EOF from 127.24.20.130:58261 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 20:14:06.496804 1227 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:14:06.590824 1230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:06.591076 1230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:06.591118 1230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:06.595161 1230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:06.595216 1230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:06.595302 1230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:14:06.600425 1230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/krb5kdc/oryx_127.24.20.131.keytab
--principal=oryx/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:42585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=46389
--tserver_master_addrs=127.24.20.190:42177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:06.601466 1230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:06.602469 1230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:06.611810 1235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:06.611855 1238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:06.611824 1236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:06.611981 1230 server_base.cc:1034] running on GCE node
I20260620 20:14:06.612401 1230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:06.613000 1230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:06.614167 1230 hybrid_clock.cc:648] HybridClock initialized: now 1781986446614153 us; error 27 us; skew 500 ppm
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986446, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:06.617027 1230 init.cc:377] Logged in from keytab as oryx/127.24.20.131@KRBTEST.COM (short username oryx)
I20260620 20:14:06.617805 1230 webserver.cc:460] Webserver started at http://127.24.20.131:46389/ using document root <none> and password file <none>
I20260620 20:14:06.618341 1230 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:06.618438 1230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:06.622220 1230 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 20:14:06.624400 1245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:06.625687 1230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:14:06.625823 1230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "d43d39368b0742b69b3e69dae35d073e"
format_stamp: "Formatted at 2026-06-20 20:14:05 on dist-test-slave-pflr"
server_key: "66c7f5936c1466ca4a33fd0505a22a06"
server_key_iv: "811483ccc8880e8029cf3d9afc14d929"
server_key_version: "encryptionkey@0"
I20260620 20:14:06.626376 1230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:06.643507 1230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:06.646378 1230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:06.646564 1230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:06.647115 1230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:06.647879 1230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:06.647945 1230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:06.648021 1230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:06.648067 1230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:06.657730 1230 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:42585
I20260620 20:14:06.657761 1358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:42585 every 8 connection(s)
I20260620 20:14:06.658672 1230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:14:06.665148 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 1230
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:14:06.670141 1097 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:06.660401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:51309 (local address 127.24.20.190:42177)
0620 20:14:06.660617 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:06.660623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:06.661404 (+ 781us) server_negotiation.cc:407] Connection header received
0620 20:14:06.662798 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:06.662801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:06.662845 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:06.662920 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:06.664094 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.664684 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.666018 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.666333 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.666460 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.670024 (+ 3564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:51309: BlockingRecv error: recv got EOF from 127.24.20.131:51309 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 20:14:06.670753 1361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:06.660818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:51309)
0620 20:14:06.661253 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:06.661294 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:06.662595 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:06.663096 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:06.663103 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:06.663453 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:06.663926 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.663939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.664823 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.664825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.665878 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.665891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.666511 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.666515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.666694 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.667297 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:06.667312 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:06.669940 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
W20260620 20:14:06.671226 1359 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:42177 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.20.190:42177: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
I20260620 20:14:06.676047 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:06.666204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.24.20.190:42177)
0620 20:14:06.666438 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:06.666442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:06.666455 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:14:06.666490 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:06.666492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:06.666527 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:06.666619 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:06.667472 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.667977 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.669097 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.669337 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.669425 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.671531 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:06.671542 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:06.671565 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:06.671599 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:06.673690 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:06.674137 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:06.674143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:06.674150 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:06.674329 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:06.674717 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:06.674721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:06.674723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:06.675081 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:06.675297 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:06.675669 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 20:14:06.675923 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:06.677753 1361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:06.671638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:42349)
0620 20:14:06.671738 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:06.671754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:06.671827 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:06.672244 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:06.672247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:06.672415 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:06.672529 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.672539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.673635 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.673638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.674151 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.674158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.674686 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.674693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.674790 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.675283 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:06.675295 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:06.677656 (+ 2361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 20:14:06.678196 1097 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:06.671699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:42349 (local address 127.24.20.190:42177)
0620 20:14:06.671807 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:06.671811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:06.671822 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:14:06.672109 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:06.672112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:06.672145 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:06.672229 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:06.672670 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.673523 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.674249 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.674551 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.674629 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.677803 (+ 3174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:42349: BlockingRecv error: recv got EOF from 127.24.20.131:42349 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:06 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:06.683964 1361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:06.678326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:44791)
0620 20:14:06.678447 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:06.678483 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:06.678564 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:06.679083 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:06.679087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:06.679253 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:06.679350 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.679360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.680135 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.680142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.681043 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:06.681055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.681530 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.681534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:06.681618 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.682064 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:06.682076 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:06.683798 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 20:14:06.684003 1097 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:06.678545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:44791 (local address 127.24.20.190:42177)
0620 20:14:06.678673 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:06.678678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:06.678696 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:14:06.678746 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:06.678749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:06.678790 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:06.678878 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:06.679523 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.680027 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.681178 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:06.681404 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:06.681532 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:06.683902 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:44791: BlockingRecv error: recv got EOF from 127.24.20.131:44791 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 20:14:06.684207 1359 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 20:14:07 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:07 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:07.311072 1364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:07.305384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:39611)
0620 20:14:07.305742 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:07.305756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:07.305854 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:07.306087 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:07.306089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:07.306271 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:07.306363 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:07.306373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:07.307091 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:07.307093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:07.307693 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:07.307701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:07.308101 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:07.308103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:07.308163 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:07.308766 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:07.308776 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:07.310891 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":133,"threads_started":1}
W20260620 20:14:07.311261 1365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:07.305501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:39611 (local address 127.24.20.190:42177)
0620 20:14:07.305770 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:07.305773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:07.305810 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:14:07.305919 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:07.305922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:07.305963 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:07.306054 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:07.306472 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:07.306986 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:07.307790 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:07.308011 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:07.308083 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:07.311094 (+ 3011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:39611: BlockingRecv error: recv got EOF from 127.24.20.129:39611 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":87,"threads_started":1}
Jun 20 20:14:07 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:07 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:07.502761 1366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:07.497396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:43627)
0620 20:14:07.497686 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:07.497699 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:07.497800 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:07.498071 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:07.498073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:07.498252 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:07.498346 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:07.498351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:07.499071 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:07.499074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:07.499682 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:07.499691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:07.500115 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:07.500118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:07.500175 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:07.500779 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:07.500790 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:07.502639 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":68,"threads_started":1}
W20260620 20:14:07.502898 1365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:07.497454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:43627 (local address 127.24.20.190:42177)
0620 20:14:07.497601 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:07.497605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:07.497766 (+ 161us) server_negotiation.cc:407] Connection header received
0620 20:14:07.497878 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:07.497880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:07.497923 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:07.497992 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:07.498459 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:07.498970 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:07.499793 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:07.500007 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:07.500100 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:07.502781 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:43627: BlockingRecv error: recv got EOF from 127.24.20.130:43627 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 20:14:07 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:07 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:07.690372 1367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:07.684861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:60695)
0620 20:14:07.685140 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:07.685155 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:07.685255 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:07.685489 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:07.685491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:07.685675 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:07.685770 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:07.685775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:07.686486 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:07.686489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:07.687122 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:07.687130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:07.687588 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:07.687591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:07.687656 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:07.688136 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:07.688148 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:07.690211 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
W20260620 20:14:07.690467 1365 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:07.684931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:60695 (local address 127.24.20.190:42177)
0620 20:14:07.685092 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:07.685097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:07.685228 (+ 131us) server_negotiation.cc:407] Connection header received
0620 20:14:07.685335 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:07.685337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:07.685383 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:07.685450 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:07.685865 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:07.686382 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:07.687228 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:07.687444 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:07.687523 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:07.690341 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:60695: BlockingRecv error: recv got EOF from 127.24.20.131:60695 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 20:14:08 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:08 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:08.317548 1368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:08.311972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:60083)
0620 20:14:08.312261 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:08.312277 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:08.312382 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:08.312765 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:08.312768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:08.312994 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:08.313089 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:08.313094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:08.313892 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:08.313895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:08.314465 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:08.314473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:08.314897 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:08.314900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:08.314964 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:08.315455 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:08.315467 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:08.317424 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
W20260620 20:14:08.317739 1369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:08.312039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:60083 (local address 127.24.20.190:42177)
0620 20:14:08.312325 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:08.312330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:08.312345 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:14:08.312487 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:08.312489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:08.312532 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:08.312644 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:08.313200 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:08.313788 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:08.314579 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:08.314800 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:08.314879 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:08.317591 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:60083: BlockingRecv error: recv got EOF from 127.24.20.129:60083 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
Jun 20 20:14:08 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:08 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:08.509462 1370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:08.503612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:35907)
0620 20:14:08.503867 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:08.503882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:08.503988 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:08.504235 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:08.504237 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:08.504417 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:08.504517 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:08.504526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:08.505326 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:08.505329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:08.505891 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:08.505900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:08.506378 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:08.506381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:08.506453 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:08.507041 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:08.507055 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:08.509293 (+ 2238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
W20260620 20:14:08.509522 1369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:08.503731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:35907 (local address 127.24.20.190:42177)
0620 20:14:08.503865 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:08.503868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:08.503961 (+ 93us) server_negotiation.cc:407] Connection header received
0620 20:14:08.504074 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:08.504076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:08.504120 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:08.504194 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:08.504670 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:08.505196 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:08.506026 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:08.506251 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:08.506343 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:08.509417 (+ 3074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:35907: BlockingRecv error: recv got EOF from 127.24.20.130:35907 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 20:14:08 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:08 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:08.696555 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:08.691203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:43767)
0620 20:14:08.691487 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:08.691503 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:08.691612 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:08.691843 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:08.691845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:08.692042 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:08.692138 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:08.692143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:08.692930 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:08.692933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:08.693546 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:08.693554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:08.693995 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:08.693998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:08.694074 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:08.694570 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:08.694581 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:08.696432 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":106,"threads_started":1}
W20260620 20:14:08.696750 1369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:08.691330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:43767 (local address 127.24.20.190:42177)
0620 20:14:08.691482 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:08.691487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:08.691582 (+ 95us) server_negotiation.cc:407] Connection header received
0620 20:14:08.691693 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:08.691695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:08.691740 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:08.691810 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:08.692235 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:08.692824 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:08.693654 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:08.693897 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:08.693985 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:08.696571 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:43767: BlockingRecv error: recv got EOF from 127.24.20.131:43767 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 20:14:09 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:09 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:09.323819 1372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:09.318394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:57199)
0620 20:14:09.318687 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:09.318701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:09.318786 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:09.319004 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:09.319006 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:09.319176 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:09.319265 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:09.319270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:09.319976 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:09.319979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:09.320642 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:09.320651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:09.321068 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:09.321071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:09.321133 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:09.321782 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:09.321799 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:09.323686 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":109,"threads_started":1}
W20260620 20:14:09.323926 1373 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:09.318446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:57199 (local address 127.24.20.190:42177)
0620 20:14:09.318749 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:09.318752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:09.318765 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:14:09.318873 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:09.318876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:09.318919 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:09.319000 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:09.319360 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:09.319873 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:09.320743 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:09.320956 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:09.321047 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:09.323806 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:57199: BlockingRecv error: recv got EOF from 127.24.20.129:57199 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":147,"threads_started":1}
Jun 20 20:14:09 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:09 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:09.516213 1374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:09.510320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:34099)
0620 20:14:09.510629 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:09.510641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:09.510782 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:09.511083 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:09.511086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:09.511319 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:09.511413 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:09.511419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:09.512225 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:09.512230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:09.512827 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:09.512835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:09.513296 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:09.513300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:09.513383 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:09.513939 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:09.513951 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:09.515997 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":77,"threads_started":1}
W20260620 20:14:09.516433 1373 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:09.510395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:34099 (local address 127.24.20.190:42177)
0620 20:14:09.510576 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:09.510581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:09.510723 (+ 142us) server_negotiation.cc:407] Connection header received
0620 20:14:09.510906 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:09.510908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:09.510952 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:09.511058 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:09.511588 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:09.512112 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:09.512946 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:09.513181 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:09.513294 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:09.516196 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:34099: BlockingRecv error: recv got EOF from 127.24.20.130:34099 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 20:14:09 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:09 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:09.703088 1375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:09.697370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:35545)
0620 20:14:09.697665 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:09.697678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:09.697779 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:09.698057 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:09.698059 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:09.698235 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:09.698329 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:09.698334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:09.699151 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:09.699156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:09.699943 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:09.699954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:09.700429 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:09.700431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:09.700536 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:09.701050 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:09.701061 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:09.702938 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
W20260620 20:14:09.703152 1373 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:09.697512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:35545 (local address 127.24.20.190:42177)
0620 20:14:09.697676 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:09.697680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:09.697751 (+ 71us) server_negotiation.cc:407] Connection header received
0620 20:14:09.697860 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:09.697863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:09.697909 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:09.697990 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:09.698438 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:09.699037 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:09.700090 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:09.700335 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:09.700413 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:09.703062 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:35545: BlockingRecv error: recv got EOF from 127.24.20.131:35545 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 20:14:10 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:10 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:10.330704 1376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:10.324713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:51627)
0620 20:14:10.324992 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:10.325006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:10.325102 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:10.325393 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:10.325396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:10.325587 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:10.325689 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:10.325696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:10.326484 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:10.326487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:10.327071 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:10.327080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:10.327703 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:10.327705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:10.327772 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:10.328259 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:10.328270 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:10.330587 (+ 2317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
W20260620 20:14:10.330883 1377 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:10.324789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:51627 (local address 127.24.20.190:42177)
0620 20:14:10.325141 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:10.325144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:10.325155 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:14:10.325189 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:10.325190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:10.325230 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:10.325324 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:10.325813 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:10.326349 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:10.327196 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:10.327419 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:10.327510 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:10.330720 (+ 3210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:51627: BlockingRecv error: recv got EOF from 127.24.20.129:51627 (error 108)
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":202,"threads_started":1}
Jun 20 20:14:10 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:10 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:10.525375 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:10.517146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:54505)
0620 20:14:10.517430 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:10.517445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:10.517579 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:10.517908 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:10.517911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:10.518093 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:10.518187 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:10.518192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:10.519048 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:10.519051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:10.519627 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:10.519635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:10.520127 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:10.520136 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:10.520255 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:10.523411 (+ 3156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:10.523423 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:10.525261 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
W20260620 20:14:10.525525 1377 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:10.517294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:54505 (local address 127.24.20.190:42177)
0620 20:14:10.517469 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:10.517473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:10.517493 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:14:10.517710 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:10.517713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:10.517762 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:10.517847 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:10.518338 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:10.518930 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:10.519745 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:10.520011 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:10.520127 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:10.525402 (+ 5275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:54505: BlockingRecv error: recv got EOF from 127.24.20.130:54505 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 20:14:10 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:10 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:10.710203 1379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:10.703890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:60127)
0620 20:14:10.704180 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:10.704194 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:10.704291 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:10.704543 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:10.704545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:10.704748 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:10.704846 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:10.704851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:10.706073 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:10.706077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:10.706783 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:10.706793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:10.707294 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:10.707297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:10.707371 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:10.707964 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:10.707978 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:10.710037 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":115,"threads_started":1}
W20260620 20:14:10.710264 1377 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:10.704025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:60127 (local address 127.24.20.190:42177)
0620 20:14:10.704170 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:10.704174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:10.704265 (+ 91us) server_negotiation.cc:407] Connection header received
0620 20:14:10.704375 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:10.704377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:10.704424 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:10.704514 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:10.705000 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:10.705896 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:10.706908 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:10.707156 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:10.707245 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:10.710153 (+ 2908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:60127: BlockingRecv error: recv got EOF from 127.24.20.131:60127 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 20:14:11 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:11 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:11.337481 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:11.331642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:47587)
0620 20:14:11.331947 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:11.331960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:11.332053 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:11.332365 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:11.332368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:11.332648 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:11.332745 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:11.332751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:11.333540 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:11.333545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:11.334126 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:11.334134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:11.334612 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:11.334615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:11.334702 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:11.335285 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:11.335297 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:11.337314 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
W20260620 20:14:11.337556 1381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:11.331704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:47587 (local address 127.24.20.190:42177)
0620 20:14:11.331936 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:11.331939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:11.332011 (+ 72us) server_negotiation.cc:407] Connection header received
0620 20:14:11.332174 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:11.332178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:11.332233 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:11.332324 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:11.332869 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:11.333427 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:11.334247 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:11.334484 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:11.334592 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:11.337439 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:47587: BlockingRecv error: recv got EOF from 127.24.20.129:47587 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":62,"threads_started":1}
Jun 20 20:14:11 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:11 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:11.532027 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:11.526245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:54823)
0620 20:14:11.526529 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:11.526542 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:11.526631 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:11.526865 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:11.526867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:11.527054 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:11.527147 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:11.527152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:11.527884 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:11.527888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:11.528477 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:11.528485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:11.529000 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:11.529003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:11.529073 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:11.529703 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:11.529715 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:11.531847 (+ 2132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
W20260620 20:14:11.532100 1381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:11.526379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:54823 (local address 127.24.20.190:42177)
0620 20:14:11.526538 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:11.526543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:11.526638 (+ 95us) server_negotiation.cc:407] Connection header received
0620 20:14:11.526703 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:11.526705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:11.526757 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:11.526830 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:11.527247 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:11.527779 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:11.528636 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:11.528855 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:11.528953 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:11.531991 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:54823: BlockingRecv error: recv got EOF from 127.24.20.130:54823 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 20:14:11 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:11 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:11.718214 1383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:11.711818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:34069)
0620 20:14:11.712079 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:11.712092 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:11.712186 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:11.712474 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:11.712476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:11.712743 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:11.712839 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:11.712846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:11.713713 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:11.713717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:11.714609 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:11.714618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:11.715222 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:11.715228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:11.715334 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:11.715932 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:11.715947 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:11.718048 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":88,"threads_started":1}
W20260620 20:14:11.718253 1381 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:11.711871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:34069 (local address 127.24.20.190:42177)
0620 20:14:11.712016 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:11.712022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:11.712163 (+ 141us) server_negotiation.cc:407] Connection header received
0620 20:14:11.712266 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:11.712269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:11.712316 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:11.712396 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:11.712971 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:11.713593 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:11.714728 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:11.715094 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:11.715215 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:11.718157 (+ 2942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:34069: BlockingRecv error: recv got EOF from 127.24.20.131:34069 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 20:14:12 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:12 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:12.343962 1384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:12.338258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:44931)
0620 20:14:12.338518 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:12.338533 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:12.338621 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:12.338898 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:12.338901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:12.339164 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:12.339286 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:12.339293 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:12.340319 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:12.340322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:12.341041 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:12.341051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:12.341452 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:12.341455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:12.341511 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:12.342035 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:12.342046 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:12.343852 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":92,"threads_started":1}
W20260620 20:14:12.344139 1385 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:12.338345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:44931 (local address 127.24.20.190:42177)
0620 20:14:12.338576 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:12.338580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:12.338596 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:14:12.338713 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:12.338715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:12.338758 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:12.338835 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:12.339436 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:12.340217 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:12.341143 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:12.341358 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:12.341435 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:12.344001 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:44931: BlockingRecv error: recv got EOF from 127.24.20.129:44931 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
Jun 20 20:14:12 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:12 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:12.538932 1386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:12.532929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:44773)
0620 20:14:12.533217 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:12.533230 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:12.533367 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:12.533735 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:12.533738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:12.534003 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:12.534164 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:12.534175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:12.535088 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:12.535091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:12.535650 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:12.535658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:12.536116 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:12.536119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:12.536182 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:12.536762 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:12.536773 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:12.538752 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":113,"threads_started":1}
W20260620 20:14:12.538993 1385 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:12.532995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:44773 (local address 127.24.20.190:42177)
0620 20:14:12.533187 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:12.533192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:12.533341 (+ 149us) server_negotiation.cc:407] Connection header received
0620 20:14:12.533538 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:12.533541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:12.533599 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:12.533709 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:12.534284 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:12.534974 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:12.535759 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:12.536000 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:12.536082 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:12.538875 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:44773: BlockingRecv error: recv got EOF from 127.24.20.130:44773 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 20:14:12 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:12 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:12.725332 1387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:12.718973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:54147)
0620 20:14:12.719255 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:12.719464 (+ 209us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:12.719570 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:12.720125 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:12.720128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:12.720369 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:12.720562 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:12.720569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:12.721362 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:12.721366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:12.721954 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:12.721962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:12.722442 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:12.722445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:12.722532 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:12.723092 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:12.723111 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:12.725162 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
W20260620 20:14:12.725397 1385 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:12.719659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:54147 (local address 127.24.20.190:42177)
0620 20:14:12.719879 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:12.719884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:12.719900 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:14:12.719952 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:12.719957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:12.720010 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:12.720085 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:12.720704 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:12.721244 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:12.722077 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:12.722313 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:12.722436 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:12.725287 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:54147: BlockingRecv error: recv got EOF from 127.24.20.131:54147 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Jun 20 20:14:13 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:13 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:13.350615 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:13.344812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:33203)
0620 20:14:13.345145 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:13.345158 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:13.345260 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:13.345554 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:13.345556 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:13.345737 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:13.345828 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:13.345833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:13.346568 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:13.346571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:13.347148 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:13.347156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:13.347598 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:13.347601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:13.347668 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:13.348292 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:13.348309 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:13.350438 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":79,"threads_started":1}
W20260620 20:14:13.350675 1389 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:13.344895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:33203 (local address 127.24.20.190:42177)
0620 20:14:13.345179 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:13.345182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:13.345212 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:14:13.345377 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:13.345380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:13.345426 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:13.345526 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:13.345924 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:13.346447 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:13.347259 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:13.347482 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:13.347569 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:13.350558 (+ 2989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:33203: BlockingRecv error: recv got EOF from 127.24.20.129:33203 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":98,"threads_started":1}
Jun 20 20:14:13 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:13 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:13.545207 1390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:13.539710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:40751)
0620 20:14:13.540030 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:13.540044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:13.540145 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:13.540372 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:13.540374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:13.540562 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:13.540717 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:13.540723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:13.541445 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:13.541448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:13.541990 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:13.541997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:13.542422 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:13.542426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:13.542540 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:13.543143 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:13.543155 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:13.545031 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":120,"threads_started":1}
W20260620 20:14:13.545341 1389 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:13.539797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:40751 (local address 127.24.20.190:42177)
0620 20:14:13.539963 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:13.539970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:13.540116 (+ 146us) server_negotiation.cc:407] Connection header received
0620 20:14:13.540225 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:13.540227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:13.540272 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:13.540338 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:13.540819 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:13.541360 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:13.542088 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:13.542298 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:13.542373 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:13.545208 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:40751: BlockingRecv error: recv got EOF from 127.24.20.130:40751 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 20:14:13 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:13 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:13.731985 1391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:13.726221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:39441)
0620 20:14:13.726567 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:13.726584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:13.726688 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:13.726934 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:13.726937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:13.727115 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:13.727211 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:13.727216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:13.727909 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:13.727912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:13.728464 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:13.728508 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:13.729055 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:13.729058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:13.729125 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:13.729733 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:13.729745 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:13.731811 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":102,"threads_started":1}
W20260620 20:14:13.732122 1389 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:13.726276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:39441 (local address 127.24.20.190:42177)
0620 20:14:13.726423 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:13.726429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:13.726658 (+ 229us) server_negotiation.cc:407] Connection header received
0620 20:14:13.726754 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:13.726756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:13.726802 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:13.726871 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:13.727313 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:13.727821 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:13.728627 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:13.728937 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:13.729021 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:13.731986 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:39441: BlockingRecv error: recv got EOF from 127.24.20.131:39441 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 20:14:14 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:14 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:14.357149 1392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:14.351414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:53287)
0620 20:14:14.351720 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:14.351736 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:14.351849 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:14.352216 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:14.352219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:14.352457 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:14.352635 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:14.352647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:14.353411 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:14.353414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:14.354012 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:14.354021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:14.354460 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:14.354463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:14.354552 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:14.355049 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:14.355060 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:14.357038 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":81,"threads_started":1}
W20260620 20:14:14.357332 1393 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:14.351541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:53287 (local address 127.24.20.190:42177)
0620 20:14:14.351798 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:14.351804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:14.351820 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:14:14.352005 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:14.352008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:14.352068 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:14.352156 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:14.352744 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:14.353304 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:14.354134 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:14.354365 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:14.354449 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:14.357165 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:53287: BlockingRecv error: recv got EOF from 127.24.20.129:53287 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
Jun 20 20:14:14 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:14 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:14:14.551772 1393 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:14.546140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:56751 (local address 127.24.20.190:42177)
0620 20:14:14.546318 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:14.546324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:14.546390 (+ 66us) server_negotiation.cc:407] Connection header received
0620 20:14:14.546611 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:14.546614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:14.546662 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:14.546756 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:14.547189 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:14.547928 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:14.548812 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:14.549057 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:14.549165 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:14.551682 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:56751: BlockingRecv error: recv got EOF from 127.24.20.130:56751 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 20:14:14.551764 1394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:14.546055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:56751)
0620 20:14:14.546317 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:14.546332 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:14.546500 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:14.546774 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:14.546777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:14.546986 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:14.547080 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:14.547085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:14.548045 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:14.548050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:14.548700 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:14.548709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:14.549175 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:14.549179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:14.549263 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:14.549717 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:14.549727 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:14.551607 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
Jun 20 20:14:14 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:14 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:14.738219 1395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:14.733047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:43953)
0620 20:14:14.733303 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:14.733316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:14.733415 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:14.733676 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:14.733678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:14.733852 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:14.733957 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:14.733962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:14.734692 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:14.734695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:14.735246 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:14.735253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:14.735665 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:14.735668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:14.735731 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:14.736207 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:14.736219 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:14.738108 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
W20260620 20:14:14.738354 1393 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:14.733081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:43953 (local address 127.24.20.190:42177)
0620 20:14:14.733216 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:14.733223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:14.733414 (+ 191us) server_negotiation.cc:407] Connection header received
0620 20:14:14.733494 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:14.733496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:14.733546 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:14.733633 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:14.734065 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:14.734570 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:14.735346 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:14.735553 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:14.735642 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:14.738237 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:43953: BlockingRecv error: recv got EOF from 127.24.20.131:43953 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 20:14:15 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:15 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:15.363641 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:15.357819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:47949)
0620 20:14:15.358125 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:15.358139 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:15.358245 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:15.358480 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:15.358482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:15.358677 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:15.358777 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:15.358782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.359499 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.359501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:15.360127 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:15.360136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.360640 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.360643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:15.360708 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:15.361245 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:15.361276 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:15.363487 (+ 2211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":111,"threads_started":1}
W20260620 20:14:15.363792 1397 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:15.357902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:47949 (local address 127.24.20.190:42177)
0620 20:14:15.358126 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:15.358130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:15.358225 (+ 95us) server_negotiation.cc:407] Connection header received
0620 20:14:15.358315 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:15.358317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:15.358363 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:15.358442 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:15.358874 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.359403 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.360252 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.360505 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.360658 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:15.363656 (+ 2998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:47949: BlockingRecv error: recv got EOF from 127.24.20.129:47949 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
Jun 20 20:14:15 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:15 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:15.561249 1398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:15.552551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:44699)
0620 20:14:15.552855 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:15.552868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:15.552963 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:15.553205 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:15.553207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:15.553380 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:15.553471 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:15.553476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.554322 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.554325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:15.554919 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:15.554927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.555380 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.555383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:15.555443 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:15.558783 (+ 3340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:15.558794 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:15.561086 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
W20260620 20:14:15.561311 1397 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:15.552693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:44699 (local address 127.24.20.190:42177)
0620 20:14:15.552841 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:15.552845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:15.552919 (+ 74us) server_negotiation.cc:407] Connection header received
0620 20:14:15.553049 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:15.553052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:15.553098 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:15.553176 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:15.553614 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.554197 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.555045 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.555283 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.555370 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:15.561206 (+ 5836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:44699: BlockingRecv error: recv got EOF from 127.24.20.130:44699 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 20:14:15 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:15 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:15.744899 1399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:15.738984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:51905)
0620 20:14:15.739239 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:15.739251 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:15.739346 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:15.739609 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:15.739612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:15.739780 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:15.739875 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:15.739880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.740751 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.740754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:15.741341 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:15.741350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.741890 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.741925 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:15.742024 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:15.742642 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:15.742659 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:15.744786 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
W20260620 20:14:15.745082 1397 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:15.739117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:51905 (local address 127.24.20.190:42177)
0620 20:14:15.739243 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:15.739246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:15.739316 (+ 70us) server_negotiation.cc:407] Connection header received
0620 20:14:15.739457 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:15.739459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:15.739503 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:15.739573 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:15.739997 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.740570 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.741491 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:15.741768 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:15.741875 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:15.744911 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:51905: BlockingRecv error: recv got EOF from 127.24.20.131:51905 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 20:14:16 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:16 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:16.370066 1400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:16.364518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:39505)
0620 20:14:16.364828 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:16.364841 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:16.364928 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:16.365202 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:16.365204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:16.365384 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:16.365505 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:16.365510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:16.366245 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:16.366247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:16.366849 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:16.366857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:16.367289 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:16.367291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:16.367398 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:16.368062 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:16.368079 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:16.369937 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":101,"threads_started":1}
W20260620 20:14:16.370232 1401 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:16.364644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:39505 (local address 127.24.20.190:42177)
0620 20:14:16.364881 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:16.364884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:16.364897 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:14:16.365044 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:16.365046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:16.365088 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:16.365164 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:16.365613 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:16.366153 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:16.366956 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:16.367187 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:16.367265 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:16.370068 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:39505: BlockingRecv error: recv got EOF from 127.24.20.129:39505 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":68,"threads_started":1}
Jun 20 20:14:16 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:16 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:16.567771 1402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:16.562132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:33897)
0620 20:14:16.562435 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:16.562449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:16.562552 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:16.562766 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:16.562768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:16.562966 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:16.563073 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:16.563079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:16.563780 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:16.563782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:16.564524 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:16.564533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:16.565045 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:16.565049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:16.565135 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:16.565713 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:16.565726 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:16.567604 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":101,"threads_started":1}
W20260620 20:14:16.567828 1401 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:16.562191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:33897 (local address 127.24.20.190:42177)
0620 20:14:16.562348 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:16.562353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:16.562503 (+ 150us) server_negotiation.cc:407] Connection header received
0620 20:14:16.562622 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:16.562625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:16.562666 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:16.562733 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:16.563160 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:16.563684 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:16.564664 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:16.564918 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:16.565044 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:16.567724 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:33897: BlockingRecv error: recv got EOF from 127.24.20.130:33897 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 20:14:16 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:16 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:16.751466 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:16.745643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:41001)
0620 20:14:16.745974 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:16.745993 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:16.746136 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:16.746425 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:16.746428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:16.746643 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:16.746767 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:16.746773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:16.747633 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:16.747638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:16.748303 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:16.748312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:16.748859 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:16.748863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:16.748943 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:16.749497 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:16.749508 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:16.751299 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
W20260620 20:14:16.751518 1401 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:16.745805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:41001 (local address 127.24.20.190:42177)
0620 20:14:16.745975 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:16.745981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:16.746099 (+ 118us) server_negotiation.cc:407] Connection header received
0620 20:14:16.746248 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:16.746252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:16.746306 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:16.746392 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:16.746887 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:16.747517 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:16.748425 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:16.748746 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:16.748856 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:16.751412 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:41001: BlockingRecv error: recv got EOF from 127.24.20.131:41001 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 20:14:17 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:17 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:17.376216 1404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:17.370773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:49547)
0620 20:14:17.371068 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:17.371088 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:17.371186 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:17.371425 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:17.371427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:17.371618 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:17.371714 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:17.371719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.372467 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.372470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:17.373160 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:17.373171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.373664 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.373668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:17.373733 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:17.374241 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:17.374252 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:17.376104 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
W20260620 20:14:17.376362 1405 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:17.370920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:49547 (local address 127.24.20.190:42177)
0620 20:14:17.371195 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:17.371199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:17.371236 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:14:17.371271 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:17.371273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:17.371317 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:17.371390 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:17.371845 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.372373 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.373270 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.373499 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.373582 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:17.376234 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:49547: BlockingRecv error: recv got EOF from 127.24.20.129:49547 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
Jun 20 20:14:17 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:17 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:17.573908 1406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:17.568502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:55673)
0620 20:14:17.568801 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:17.568814 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:17.568908 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:17.569159 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:17.569165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:17.569350 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:17.569446 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:17.569451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.570181 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.570184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:17.570795 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:17.570804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.571270 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.571273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:17.571335 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:17.571856 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:17.571868 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:17.573777 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":133,"threads_started":1}
W20260620 20:14:17.574050 1405 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:17.568575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:55673 (local address 127.24.20.190:42177)
0620 20:14:17.568734 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:17.568737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:17.568879 (+ 142us) server_negotiation.cc:407] Connection header received
0620 20:14:17.568994 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:17.568996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:17.569039 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:17.569105 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:17.569547 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.570080 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.570938 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.571166 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.571257 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:17.573917 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:55673: BlockingRecv error: recv got EOF from 127.24.20.130:55673 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 20:14:17 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:17 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:17.757848 1407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:17.752275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:38003)
0620 20:14:17.752576 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:17.752630 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:17.752760 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:17.753073 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:17.753076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:17.753254 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:17.753348 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:17.753353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.754182 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.754185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:17.754774 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:17.754782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.755188 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.755190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:17.755248 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:17.755715 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:17.755726 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:17.757653 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":91,"threads_started":1}
W20260620 20:14:17.757966 1405 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:17.752328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:38003 (local address 127.24.20.190:42177)
0620 20:14:17.752484 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:17.752489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:17.752706 (+ 217us) server_negotiation.cc:407] Connection header received
0620 20:14:17.752866 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:17.752869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:17.752925 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:17.753035 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:17.753449 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.754077 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.754876 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:17.755087 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:17.755160 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:17.757853 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:38003: BlockingRecv error: recv got EOF from 127.24.20.131:38003 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 20:14:18 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:18 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:18.382583 1408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:18.377078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:47861)
0620 20:14:18.377383 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:18.377397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:18.377488 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:18.377738 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:18.377740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:18.377931 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:18.378033 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:18.378039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.378743 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.378746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:18.379301 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:18.379308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.379741 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.379743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:18.379802 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:18.380273 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:18.380284 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:18.382387 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":73,"threads_started":1}
W20260620 20:14:18.382640 1409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:18.377148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:47861 (local address 127.24.20.190:42177)
0620 20:14:18.377437 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:18.377440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:18.377455 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:14:18.377577 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:18.377580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:18.377624 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:18.377699 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:18.378132 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.378651 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.379408 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.379643 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.379724 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:18.382520 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:47861: BlockingRecv error: recv got EOF from 127.24.20.129:47861 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":116,"threads_started":1}
Jun 20 20:14:18 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:18 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:18.580490 1410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:18.574818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:33297)
0620 20:14:18.575108 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:18.575123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:18.575225 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:18.575437 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:18.575439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:18.575626 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:18.575718 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:18.575723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.576836 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.576839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:18.577589 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:18.577599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.578002 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.578004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:18.578063 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:18.578604 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:18.578616 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:18.580376 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
W20260620 20:14:18.580703 1409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:18.574881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:33297 (local address 127.24.20.190:42177)
0620 20:14:18.575036 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:18.575043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:18.575193 (+ 150us) server_negotiation.cc:407] Connection header received
0620 20:14:18.575290 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:18.575293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:18.575336 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:18.575400 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:18.575836 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.576705 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.577696 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.577908 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.577987 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:18.580514 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:33297: BlockingRecv error: recv got EOF from 127.24.20.130:33297 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 20:14:18 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:18 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:18.764259 1411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:18.758706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:41731)
0620 20:14:18.759060 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:18.759073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:18.759208 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:18.759471 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:18.759474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:18.759675 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:18.759766 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:18.759772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.760519 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.760524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:18.761161 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:18.761169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.761627 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.761631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:18.761714 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:18.762277 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:18.762289 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:18.764148 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":103,"threads_started":1}
W20260620 20:14:18.764470 1409 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:18.758758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:41731 (local address 127.24.20.190:42177)
0620 20:14:18.759026 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:18.759030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:18.759150 (+ 120us) server_negotiation.cc:407] Connection header received
0620 20:14:18.759305 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:18.759310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:18.759365 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:18.759448 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:18.759881 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.760412 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.761282 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:18.761516 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:18.761626 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:18.764329 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:41731: BlockingRecv error: recv got EOF from 127.24.20.131:41731 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 20:14:19 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:19 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:19.388419 1412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.383348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:45703)
0620 20:14:19.383611 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.383624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:19.383707 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:19.383987 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:19.383989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:19.384166 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:19.384254 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.384259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.385087 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.385090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.385637 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.385645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.386036 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.386039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.386098 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.386574 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:19.386585 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:19.388308 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
W20260620 20:14:19.388558 1413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:19.383407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:45703 (local address 127.24.20.190:42177)
0620 20:14:19.383647 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:19.383650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:19.383666 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:14:19.383819 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:19.383822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:19.383869 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:19.383959 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:19.384375 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.384986 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.385732 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.385945 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.386018 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.388433 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:45703: BlockingRecv error: recv got EOF from 127.24.20.129:45703 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
Jun 20 20:14:19 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:19 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:19.587208 1414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.581248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:55945)
0620 20:14:19.581521 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.581536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:19.581627 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:19.581853 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:19.581855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:19.582051 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:19.582149 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.582155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.582982 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.582986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.583550 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.583560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.584046 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.584048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.584111 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.584906 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:19.584936 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:19.587086 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":96,"threads_started":1}
W20260620 20:14:19.587441 1413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:19.581424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:55945 (local address 127.24.20.190:42177)
0620 20:14:19.581638 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:19.581642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:19.581665 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:14:19.581701 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:19.581703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:19.581743 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:19.581815 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:19.582270 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.582849 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.583663 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.583876 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.584036 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.587305 (+ 3269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:55945: BlockingRecv error: recv got EOF from 127.24.20.130:55945 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 20:14:19 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:19 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:19.770498 1415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:19.764974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:36199)
0620 20:14:19.765297 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:19.765319 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:19.765498 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:19.765784 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:19.765787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:19.766007 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:19.766098 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.766103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.766857 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.766861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.767436 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:19.767443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.767913 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.767917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:19.768003 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.768549 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:19.768561 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:19.770382 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":82,"threads_started":1}
W20260620 20:14:19.770630 1413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:19.765078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:36199 (local address 127.24.20.190:42177)
0620 20:14:19.765252 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:19.765257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:19.765395 (+ 138us) server_negotiation.cc:407] Connection header received
0620 20:14:19.765617 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:19.765620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:19.765676 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:19.765760 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:19.766210 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.766741 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.767554 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:19.767805 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:19.767912 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:19.770517 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:36199: BlockingRecv error: recv got EOF from 127.24.20.131:36199 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 20:14:20 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:20 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:20.394750 1416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:20.389278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:58267)
0620 20:14:20.389563 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:20.389582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:20.389686 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:20.389925 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:20.389927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:20.390104 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:20.390193 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:20.390198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.390964 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.390966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:20.391527 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:20.391534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.391972 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.391975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:20.392033 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:20.392542 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:20.392553 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:20.394569 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":69,"threads_started":1}
W20260620 20:14:20.394806 1417 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:20.389364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:58267 (local address 127.24.20.190:42177)
0620 20:14:20.389719 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:20.389723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:20.389736 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:14:20.389771 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:20.389773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:20.389813 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:20.389896 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:20.390290 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.390866 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.391648 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.391872 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.391961 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:20.394684 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:58267: BlockingRecv error: recv got EOF from 127.24.20.129:58267 (error 108)
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":73,"threads_started":1}
Jun 20 20:14:20 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:20 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:20.598169 1418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:20.587922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:33175)
0620 20:14:20.588200 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:20.588223 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:20.588349 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:20.588690 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:20.588693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:20.588909 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:20.589009 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:20.589015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.589767 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.589772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:20.590366 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:20.590373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.590823 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.590827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:20.590912 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:20.596181 (+ 5269us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:20.596193 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:20.597992 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
W20260620 20:14:20.598206 1417 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:20.587991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:33175 (local address 127.24.20.190:42177)
0620 20:14:20.588156 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:20.588160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:20.588298 (+ 138us) server_negotiation.cc:407] Connection header received
0620 20:14:20.588458 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:20.588462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:20.588543 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:20.588667 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:20.589125 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.589661 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.590493 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.590715 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.590821 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:20.598107 (+ 7286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:33175: BlockingRecv error: recv got EOF from 127.24.20.130:33175 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 20:14:20 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:20 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.131@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:20.776741 1419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:20.771248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.131:43203)
0620 20:14:20.771521 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:20.771534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:20.771641 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:20.771892 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:20.771894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:20.772062 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:20.772155 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:20.772160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.772997 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.773000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:20.773553 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:20.773560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.773972 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.773974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:20.774030 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:20.774521 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:20.774531 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:20.776538 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
W20260620 20:14:20.776798 1417 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:20.771300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:43203 (local address 127.24.20.190:42177)
0620 20:14:20.771444 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:20.771449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:20.771608 (+ 159us) server_negotiation.cc:407] Connection header received
0620 20:14:20.771724 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:20.771726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:20.771775 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:20.771847 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:20.772253 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.772900 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.773661 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:20.773884 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:20.773959 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:20.776687 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.131:43203: BlockingRecv error: recv got EOF from 127.24.20.131:43203 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 20:14:21 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:21 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.129@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:21.400823 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.395563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.129:33485)
0620 20:14:21.395862 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:21.395875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:21.395958 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:21.396228 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:21.396230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:21.396413 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:21.396504 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.396509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.397270 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.397273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.397851 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.397860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.398318 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.398321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.398387 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.398905 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:21.398916 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:21.400709 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
W20260620 20:14:21.400972 1421 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:21.395681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:33485 (local address 127.24.20.190:42177)
0620 20:14:21.395906 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:21.395910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:21.395922 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:14:21.396066 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:21.396069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:21.396113 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:21.396195 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:21.396640 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.397171 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.397977 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.398211 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.398290 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.400840 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.129:33485: BlockingRecv error: recv got EOF from 127.24.20.129:33485 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":55,"threads_started":1}
Jun 20 20:14:21 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:14:21 dist-test-slave-pflr krb5kdc[363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.20.130@KRBTEST.COM for oryx/127.24.20.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:14:21.604768 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.599101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42177 (local address 127.24.20.130:42091)
0620 20:14:21.599398 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:21.599414 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:21.599567 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:21.599891 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:21.599894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:21.600172 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:21.600296 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.600303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.601070 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.601072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.601747 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:21.601756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.602150 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.602153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:21.602219 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.602724 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:21.602735 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:21.604656 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.20.190:42177: Server oryx/127.24.20.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":109,"threads_started":1}
W20260620 20:14:21.605034 1421 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:21.599251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:42091 (local address 127.24.20.190:42177)
0620 20:14:21.599385 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:21.599389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:21.599562 (+ 173us) server_negotiation.cc:407] Connection header received
0620 20:14:21.599684 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:21.599688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:21.599746 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:21.599829 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:21.600407 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.600975 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.601847 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.602057 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.602130 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.604789 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.20.130:42091: BlockingRecv error: recv got EOF from 127.24.20.130:42091 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:14:21.667553 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 946
I20260620 20:14:21.673143 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 1098
I20260620 20:14:21.678130 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 1230
I20260620 20:14:21.682896 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 876
2026-06-20T20:14:21Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17934 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T20:14:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:21Z Disabled control of system clock
I20260620 20:14:21.711751 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:42375
--webserver_interface=127.24.20.190
--webserver_port=0
--builtin_ntp_servers=127.24.20.148:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:42375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 20:14:21.819705 1426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:21.819952 1426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:21.819986 1426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:21.823984 1426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:21.824040 1426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:21.824056 1426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:21.824074 1426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:21.824087 1426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:21.828866 1426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:42375
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:42375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:21.829865 1426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:21.831171 1426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:14:21.839347 1426 server_base.cc:1034] running on GCE node
W20260620 20:14:21.839313 1432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:21.839332 1431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:21.839313 1434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:21.840092 1426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:21.841267 1426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:21.842456 1426 hybrid_clock.cc:648] HybridClock initialized: now 1781986461842442 us; error 27 us; skew 500 ppm
I20260620 20:14:21.844198 1426 webserver.cc:460] Webserver started at http://127.24.20.190:35291/ using document root <none> and password file <none>
I20260620 20:14:21.844813 1426 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:21.844900 1426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:21.845124 1426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:21.847150 1426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "5f934ea5a59b46e8b49821d850745c6a"
format_stamp: "Formatted at 2026-06-20 20:14:21 on dist-test-slave-pflr"
server_key: "abf7c61f88bd42dcd56177899406c22a"
server_key_iv: "6dc2a0fd05ae4d7a65c35cfad9465b7e"
server_key_version: "encryptionkey@0"
I20260620 20:14:21.847707 1426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "5f934ea5a59b46e8b49821d850745c6a"
format_stamp: "Formatted at 2026-06-20 20:14:21 on dist-test-slave-pflr"
server_key: "abf7c61f88bd42dcd56177899406c22a"
server_key_iv: "6dc2a0fd05ae4d7a65c35cfad9465b7e"
server_key_version: "encryptionkey@0"
I20260620 20:14:21.851171 1426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:14:21.853488 1440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:21.854596 1426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:14:21.854738 1426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "5f934ea5a59b46e8b49821d850745c6a"
format_stamp: "Formatted at 2026-06-20 20:14:21 on dist-test-slave-pflr"
server_key: "abf7c61f88bd42dcd56177899406c22a"
server_key_iv: "6dc2a0fd05ae4d7a65c35cfad9465b7e"
server_key_version: "encryptionkey@0"
I20260620 20:14:21.854844 1426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:21.871320 1426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:21.871917 1426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:21.872095 1426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:21.878681 1426 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:42375
I20260620 20:14:21.878693 1492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:42375 every 8 connection(s)
I20260620 20:14:21.879532 1426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:21.882539 1493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:21.886925 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 1426
I20260620 20:14:21.887068 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:14:21.887331 24658 external_mini_cluster.cc:1428] Setting key 81ddec35a29768f6ff4b5da3be2ce800
I20260620 20:14:21.887823 1493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a: Bootstrap starting.
I20260620 20:14:21.890272 1493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:21.890977 1493 log.cc:824] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:21.892778 1493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a: No bootstrap required, opened a new log
I20260620 20:14:21.896322 1493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f934ea5a59b46e8b49821d850745c6a" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42375 } }
I20260620 20:14:21.896471 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:21.888992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.24.20.190:42375)
0620 20:14:21.889768 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:21.889781 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:21.889813 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:14:21.890477 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:21.890494 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:21.890858 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:21.891181 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:21.891569 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.892479 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.893435 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:21.893882 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:21.894060 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:21.894896 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:21.894901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:21.894913 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:21.895270 (+ 357us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:21.895320 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:21.895339 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:21.895444 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:21.895585 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 20:14:21.895814 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":122,"threads_started":1}
I20260620 20:14:21.896553 1493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:21.896663 1493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f934ea5a59b46e8b49821d850745c6a, State: Initialized, Role: FOLLOWER
I20260620 20:14:21.897167 1493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f934ea5a59b46e8b49821d850745c6a" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42375 } }
I20260620 20:14:21.897315 1493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:21.897374 1493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:21.897462 1493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:21.898255 1493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f934ea5a59b46e8b49821d850745c6a" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42375 } }
I20260620 20:14:21.898592 1493 leader_election.cc:302] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f934ea5a59b46e8b49821d850745c6a; no voters:
I20260620 20:14:21.898834 1493 leader_election.cc:288] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:21.898947 1498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:21.899214 1498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [term 1 LEADER]: Becoming Leader. State: Replica: 5f934ea5a59b46e8b49821d850745c6a, State: Running, Role: LEADER
I20260620 20:14:21.899490 1498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f934ea5a59b46e8b49821d850745c6a" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42375 } }
I20260620 20:14:21.899916 1493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:21.901099 1500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f934ea5a59b46e8b49821d850745c6a. Latest consensus state: current_term: 1 leader_uuid: "5f934ea5a59b46e8b49821d850745c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f934ea5a59b46e8b49821d850745c6a" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42375 } } }
I20260620 20:14:21.901124 1499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f934ea5a59b46e8b49821d850745c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f934ea5a59b46e8b49821d850745c6a" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42375 } } }
I20260620 20:14:21.901232 1500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:21.901233 1499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:21.901580 1507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:21.904249 1507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:21.909056 1507 catalog_manager.cc:1347] Generated new cluster ID: cc4c7da429ad40babae6d0fc62618e26
I20260620 20:14:21.909143 1507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:21.915761 1507 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:21.916522 1507 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:21.926131 1507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f934ea5a59b46e8b49821d850745c6a: Generated new TSK 0
I20260620 20:14:21.926852 1507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:21.930868 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42375
--builtin_ntp_servers=127.24.20.148:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 20:14:22.039621 1517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:22.039856 1517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:22.039892 1517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:22.043900 1517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:22.043958 1517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:22.044013 1517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:14:22.048838 1517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:22.049850 1517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:22.051179 1517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:22.060432 1523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:22.060431 1522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:22.060624 1517 server_base.cc:1034] running on GCE node
W20260620 20:14:22.060518 1525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:22.061179 1517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:22.061769 1517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:22.062929 1517 hybrid_clock.cc:648] HybridClock initialized: now 1781986462062898 us; error 42 us; skew 500 ppm
I20260620 20:14:22.064695 1517 webserver.cc:460] Webserver started at http://127.24.20.129:39127/ using document root <none> and password file <none>
I20260620 20:14:22.065238 1517 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:22.065299 1517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:22.065527 1517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:22.067448 1517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "3d0adb14c2ff4da791968ee0f4f2987b"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "e26088309ccab1e00baa919e67744973"
server_key_iv: "014af39e58bccd71779186e6cc3bb147"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.068012 1517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "3d0adb14c2ff4da791968ee0f4f2987b"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "e26088309ccab1e00baa919e67744973"
server_key_iv: "014af39e58bccd71779186e6cc3bb147"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.071723 1517 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:14:22.074280 1531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:22.075430 1517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:14:22.075564 1517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "3d0adb14c2ff4da791968ee0f4f2987b"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "e26088309ccab1e00baa919e67744973"
server_key_iv: "014af39e58bccd71779186e6cc3bb147"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.075704 1517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:22.087213 1517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:22.087816 1517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:22.087999 1517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:22.088528 1517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:22.089411 1517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:22.089483 1517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:22.089550 1517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:22.089608 1517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:22.097759 1517 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:34805
I20260620 20:14:22.097782 1644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:34805 every 8 connection(s)
I20260620 20:14:22.098592 1517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:14:22.106782 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 1517
I20260620 20:14:22.106884 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:14:22.107211 24658 external_mini_cluster.cc:1428] Setting key c84aa21ab6e09bca2180bbb44d5e6359
I20260620 20:14:22.108074 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.100489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:45203 (local address 127.24.20.190:42375)
0620 20:14:22.100627 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:22.100632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:22.101585 (+ 953us) server_negotiation.cc:407] Connection header received
0620 20:14:22.103066 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:22.103069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:22.103136 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:22.103242 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:22.104110 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.104731 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.105845 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.106072 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.106278 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.107551 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:22.107556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:22.107561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:22.107578 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:22.107600 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:22.107611 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:22.107683 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:22.107878 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 20:14:22.107938 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 20:14:22.108981 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.100727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42375 (local address 127.24.20.129:45203)
0620 20:14:22.101427 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.101478 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:22.102850 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:22.103404 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:22.103410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:22.103490 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:22.103957 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:22.103969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.104849 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.104852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:22.105708 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:22.105717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.106301 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.106307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:22.106474 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.107220 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:22.107222 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:22.107259 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:22.107351 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:22.107709 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:22.107718 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:22.107723 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:14:22.107867 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 20:14:22.108113 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
I20260620 20:14:22.109344 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42375
--builtin_ntp_servers=127.24.20.148:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 20:14:22.111181 1645 heartbeater.cc:347] Connected to a master server at 127.24.20.190:42375
I20260620 20:14:22.111397 1645 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:22.112006 1645 heartbeater.cc:511] Master 127.24.20.190:42375 requested a full tablet report, sending...
I20260620 20:14:22.113533 1457 ts_manager.cc:194] Registered new tserver with Master: 3d0adb14c2ff4da791968ee0f4f2987b (127.24.20.129:34805)
I20260620 20:14:22.114632 1457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.20.129:45203
W20260620 20:14:22.219486 1648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:22.219726 1648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:22.219762 1648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:22.223717 1648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:22.223767 1648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:22.223858 1648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:14:22.228399 1648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:22.229386 1648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:22.230690 1648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:14:22.239899 1648 server_base.cc:1034] running on GCE node
W20260620 20:14:22.239916 1653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:22.239864 1654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:22.240072 1656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:22.240419 1648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:22.241068 1648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:22.242233 1648 hybrid_clock.cc:648] HybridClock initialized: now 1781986462242217 us; error 28 us; skew 500 ppm
I20260620 20:14:22.243990 1648 webserver.cc:460] Webserver started at http://127.24.20.130:44295/ using document root <none> and password file <none>
I20260620 20:14:22.244565 1648 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:22.244705 1648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:22.244946 1648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:22.246889 1648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "30b44cce67814ac1a017f2ea491ab38f"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "74df00c1cb748f672a5e939f7e769dd2"
server_key_iv: "322ea1eeb407804b5c6b7d56bdf9e61e"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.247449 1648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "30b44cce67814ac1a017f2ea491ab38f"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "74df00c1cb748f672a5e939f7e769dd2"
server_key_iv: "322ea1eeb407804b5c6b7d56bdf9e61e"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.250957 1648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:14:22.253410 1662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:22.254580 1648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:14:22.254709 1648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "30b44cce67814ac1a017f2ea491ab38f"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "74df00c1cb748f672a5e939f7e769dd2"
server_key_iv: "322ea1eeb407804b5c6b7d56bdf9e61e"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.254817 1648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:22.271610 1648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:22.272235 1648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:22.272419 1648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:22.272970 1648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:22.273835 1648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:22.273904 1648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:22.273974 1648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:22.274019 1648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:22.282372 1648 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:40009
I20260620 20:14:22.282397 1775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:40009 every 8 connection(s)
I20260620 20:14:22.283241 1648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:14:22.285596 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 1648
I20260620 20:14:22.285718 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:14:22.286002 24658 external_mini_cluster.cc:1428] Setting key 5ef52aebe15ea54d0074b9b5545cb7f8
I20260620 20:14:22.287978 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42375
--builtin_ntp_servers=127.24.20.148:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 20:14:22.293431 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.285027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:41337 (local address 127.24.20.190:42375)
0620 20:14:22.285163 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:22.285167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:22.286263 (+ 1096us) server_negotiation.cc:407] Connection header received
0620 20:14:22.287811 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:22.287817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:22.287877 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:22.287969 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:22.288966 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.289633 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.291082 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.291495 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.291618 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.292828 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:22.292836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:22.292842 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:22.292859 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:22.292887 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:22.292900 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:22.292985 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:22.293139 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 20:14:22.293214 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:14:22.294135 1778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.285415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42375 (local address 127.24.20.130:41337)
0620 20:14:22.286095 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.286145 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:22.287614 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:22.288169 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:22.288182 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:22.288268 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:22.288831 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:22.288844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.289769 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.289775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:22.290955 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:22.290965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.291718 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.291723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:22.291877 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.292519 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:22.292520 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:22.292558 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:22.292677 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:22.292998 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:22.293005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:22.293009 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:14:22.293100 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 20:14:22.293314 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":130,"threads_started":1}
I20260620 20:14:22.295418 1776 heartbeater.cc:347] Connected to a master server at 127.24.20.190:42375
I20260620 20:14:22.295672 1776 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:22.296166 1776 heartbeater.cc:511] Master 127.24.20.190:42375 requested a full tablet report, sending...
I20260620 20:14:22.297196 1457 ts_manager.cc:194] Registered new tserver with Master: 30b44cce67814ac1a017f2ea491ab38f (127.24.20.130:40009)
I20260620 20:14:22.297757 1457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.20.130:41337
W20260620 20:14:22.396867 1779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:22.397135 1779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:22.397197 1779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:22.401328 1779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:22.401422 1779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:22.401535 1779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:14:22.406328 1779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:22.407332 1779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:22.408727 1779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:22.417713 1785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:22.417708 1787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:22.417955 1779 server_base.cc:1034] running on GCE node
W20260620 20:14:22.417708 1784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:22.418394 1779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:22.418998 1779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:22.420156 1779 hybrid_clock.cc:648] HybridClock initialized: now 1781986462420140 us; error 25 us; skew 500 ppm
I20260620 20:14:22.421904 1779 webserver.cc:460] Webserver started at http://127.24.20.131:38095/ using document root <none> and password file <none>
I20260620 20:14:22.422461 1779 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:22.422523 1779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:22.422750 1779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:22.424739 1779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "099f9ec9258a4e0d92d2442c6e536ad0"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "2353c2c90de64b91bb0e9148db88f00e"
server_key_iv: "c3442f27ded31cf3ea70e6a983a55a56"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.425266 1779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "099f9ec9258a4e0d92d2442c6e536ad0"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "2353c2c90de64b91bb0e9148db88f00e"
server_key_iv: "c3442f27ded31cf3ea70e6a983a55a56"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.428790 1779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:14:22.431078 1793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:22.432196 1779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:14:22.432314 1779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "099f9ec9258a4e0d92d2442c6e536ad0"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "2353c2c90de64b91bb0e9148db88f00e"
server_key_iv: "c3442f27ded31cf3ea70e6a983a55a56"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.432408 1779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:22.454768 1779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:22.455407 1779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:22.455600 1779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:22.456166 1779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:22.457159 1779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:22.457218 1779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:22.457286 1779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:22.457321 1779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:22.466321 1779 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:45375
I20260620 20:14:22.466349 1906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:45375 every 8 connection(s)
I20260620 20:14:22.467182 1779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:14:22.474327 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 1779
I20260620 20:14:22.474409 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:14:22.474740 24658 external_mini_cluster.cc:1428] Setting key 0979e8e327cc61bb9124bb62f1a2da24
I20260620 20:14:22.475931 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.468933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:57545 (local address 127.24.20.190:42375)
0620 20:14:22.469085 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:22.469091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:22.470120 (+ 1029us) server_negotiation.cc:407] Connection header received
0620 20:14:22.471417 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:22.471420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:22.471464 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:22.471537 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:22.472369 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.472896 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.473948 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.474174 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.474285 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.475452 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:22.475455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:22.475457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:22.475467 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:22.475485 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:22.475493 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:22.475577 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:22.475743 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 20:14:22.475798 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:14:22.476845 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.469352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42375 (local address 127.24.20.131:57545)
0620 20:14:22.469967 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.470004 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:22.471237 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:22.471711 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:22.471717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:22.471797 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:22.472253 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:22.472264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.472995 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.472998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:22.473850 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:22.473858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:22.474325 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:22.474329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:22.474555 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:22.475214 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:22.475216 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:22.475253 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:22.475310 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:22.475578 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:22.475589 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:22.475597 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:14:22.475743 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 20:14:22.475999 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
I20260620 20:14:22.477804 1907 heartbeater.cc:347] Connected to a master server at 127.24.20.190:42375
I20260620 20:14:22.478025 1907 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:22.478495 1907 heartbeater.cc:511] Master 127.24.20.190:42375 requested a full tablet report, sending...
I20260620 20:14:22.479561 1457 ts_manager.cc:194] Registered new tserver with Master: 099f9ec9258a4e0d92d2442c6e536ad0 (127.24.20.131:45375)
I20260620 20:14:22.480181 1457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.20.131:57545
I20260620 20:14:22.488308 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 20:14:22.490900 1496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:22.490157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55276 (local address 127.24.20.190:42375)
0620 20:14:22.490270 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:22.490273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:22.490413 (+ 140us) server_negotiation.cc:407] Connection header received
0620 20:14:22.490507 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:22.490509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:22.490542 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:22.490629 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:22.490818 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55276: BlockingRecv error: recv got EOF from 127.0.0.1:55276 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:14:22.491159 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 1517
I20260620 20:14:22.496486 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 1648
I20260620 20:14:22.501534 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 1779
I20260620 20:14:22.506500 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 1426
2026-06-20T20:14:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (822 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T20:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:22Z Disabled control of system clock
I20260620 20:14:22.534945 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:44465
--webserver_interface=127.24.20.190
--webserver_port=0
--builtin_ntp_servers=127.24.20.148:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:44465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 20:14:22.641338 1918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:22.641579 1918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:22.641614 1918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:22.645569 1918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:22.645624 1918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:22.645640 1918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:22.645656 1918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:22.645671 1918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:22.650454 1918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:44465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:44465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:22.651525 1918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:22.652931 1918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:22.662253 1923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:22.662286 1926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:22.662806 1924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:22.662786 1918 server_base.cc:1034] running on GCE node
I20260620 20:14:22.663755 1918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:22.665519 1918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:22.666692 1918 hybrid_clock.cc:648] HybridClock initialized: now 1781986462666636 us; error 71 us; skew 500 ppm
I20260620 20:14:22.669484 1918 webserver.cc:460] Webserver started at http://127.24.20.190:41515/ using document root <none> and password file <none>
I20260620 20:14:22.670406 1918 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:22.670537 1918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:22.670837 1918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:22.673704 1918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "5a7ac8acbaf64cd78de7cb2faf9067e0"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "d8d899457eae0ad15f0fb5b964fbe7f6"
server_key_iv: "cff4598a33f5c5c25dad75d9e4e45912"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.674346 1918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "5a7ac8acbaf64cd78de7cb2faf9067e0"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "d8d899457eae0ad15f0fb5b964fbe7f6"
server_key_iv: "cff4598a33f5c5c25dad75d9e4e45912"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.678930 1918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:14:22.682025 1932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:22.683602 1918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:14:22.683730 1918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "5a7ac8acbaf64cd78de7cb2faf9067e0"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "d8d899457eae0ad15f0fb5b964fbe7f6"
server_key_iv: "cff4598a33f5c5c25dad75d9e4e45912"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.683847 1918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:22.690395 1918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:22.691243 1918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:22.691417 1918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:22.699187 1918 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:44465
I20260620 20:14:22.699231 1984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:44465 every 8 connection(s)
I20260620 20:14:22.700235 1918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:22.703527 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:22.709348 1985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0: Bootstrap starting.
I20260620 20:14:22.710220 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 1918
I20260620 20:14:22.710310 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:14:22.710624 24658 external_mini_cluster.cc:1428] Setting key f2f2b36f548420fb75259f934ed1cddc
I20260620 20:14:22.712000 1985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:22.712770 1985 log.cc:824] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:22.714707 1985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0: No bootstrap required, opened a new log
I20260620 20:14:22.715767 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.711738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.24.20.190:44465)
0620 20:14:22.712378 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:22.712394 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:22.712426 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:14:22.713183 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:22.713201 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:22.713496 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:22.713823 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:22.714591 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:22.714595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:22.714606 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:22.714617 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:22.714669 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:22.714684 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:22.714797 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:22.714899 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 20:14:22.715131 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":143,"threads_started":1}
I20260620 20:14:22.718411 1985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7ac8acbaf64cd78de7cb2faf9067e0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44465 } }
I20260620 20:14:22.718641 1985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:22.718719 1985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a7ac8acbaf64cd78de7cb2faf9067e0, State: Initialized, Role: FOLLOWER
I20260620 20:14:22.719118 1985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7ac8acbaf64cd78de7cb2faf9067e0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44465 } }
I20260620 20:14:22.719264 1985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:22.719348 1985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:22.719470 1985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:22.720357 1985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7ac8acbaf64cd78de7cb2faf9067e0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44465 } }
I20260620 20:14:22.720733 1985 leader_election.cc:302] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a7ac8acbaf64cd78de7cb2faf9067e0; no voters:
I20260620 20:14:22.721055 1985 leader_election.cc:288] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:22.721182 1990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:22.721377 1990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [term 1 LEADER]: Becoming Leader. State: Replica: 5a7ac8acbaf64cd78de7cb2faf9067e0, State: Running, Role: LEADER
I20260620 20:14:22.721678 1990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7ac8acbaf64cd78de7cb2faf9067e0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44465 } }
I20260620 20:14:22.722245 1985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:22.723392 1991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a7ac8acbaf64cd78de7cb2faf9067e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7ac8acbaf64cd78de7cb2faf9067e0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44465 } } }
I20260620 20:14:22.723439 1992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a7ac8acbaf64cd78de7cb2faf9067e0. Latest consensus state: current_term: 1 leader_uuid: "5a7ac8acbaf64cd78de7cb2faf9067e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a7ac8acbaf64cd78de7cb2faf9067e0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44465 } } }
I20260620 20:14:22.723515 1991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:22.723546 1992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:22.723892 1999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:22.726835 1999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:22.731683 1999 catalog_manager.cc:1347] Generated new cluster ID: e8f331d741ab4040a0cb73e6fff59057
I20260620 20:14:22.731750 1999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:22.746341 1999 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:22.746662 1999 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:22.751245 1999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a7ac8acbaf64cd78de7cb2faf9067e0: Generated new TSK 0
I20260620 20:14:22.751977 1999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:14:22.756716 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44465
--builtin_ntp_servers=127.24.20.148:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 20:14:22.866242 2009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:22.866485 2009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:22.866554 2009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:22.870564 2009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:22.870617 2009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:22.870729 2009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:14:22.875349 2009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:22.876333 2009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:22.877720 2009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:22.886893 2014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:22.886924 2009 server_base.cc:1034] running on GCE node
W20260620 20:14:22.886950 2015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:22.886950 2017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:22.887512 2009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:22.888110 2009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:22.889283 2009 hybrid_clock.cc:648] HybridClock initialized: now 1781986462889257 us; error 39 us; skew 500 ppm
I20260620 20:14:22.890954 2009 webserver.cc:460] Webserver started at http://127.24.20.129:36883/ using document root <none> and password file <none>
I20260620 20:14:22.891491 2009 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:22.891579 2009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:22.891803 2009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:22.893836 2009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "d84b79b6c09d4cc2a575f3a8cdde52c8"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "2c8b1ce20795e02c38ceb9ac82ad9d3e"
server_key_iv: "2c77db4160f41cc3a8f119e2e431e95d"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.894367 2009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "d84b79b6c09d4cc2a575f3a8cdde52c8"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "2c8b1ce20795e02c38ceb9ac82ad9d3e"
server_key_iv: "2c77db4160f41cc3a8f119e2e431e95d"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.897761 2009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:14:22.900099 2023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:22.901237 2009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:14:22.901362 2009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "d84b79b6c09d4cc2a575f3a8cdde52c8"
format_stamp: "Formatted at 2026-06-20 20:14:22 on dist-test-slave-pflr"
server_key: "2c8b1ce20795e02c38ceb9ac82ad9d3e"
server_key_iv: "2c77db4160f41cc3a8f119e2e431e95d"
server_key_version: "encryptionkey@0"
I20260620 20:14:22.901466 2009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:22.905824 2009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:22.906373 2009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:22.906549 2009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:22.907060 2009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:22.907902 2009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:22.907971 2009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:22.908034 2009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:22.908082 2009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:22.916509 2009 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:45615
I20260620 20:14:22.916561 2136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:45615 every 8 connection(s)
I20260620 20:14:22.917373 2009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:14:22.923221 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.919047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:55839 (local address 127.24.20.190:44465)
0620 20:14:22.919220 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:22.919225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:22.920198 (+ 973us) server_negotiation.cc:407] Connection header received
0620 20:14:22.921450 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:22.921452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:22.921488 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:22.921561 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:22.922702 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:22.922711 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:22.922714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:22.922723 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:22.922741 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:22.922749 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:22.922820 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:22.923009 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 20:14:22.923044 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:14:22.924136 2139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:22.919426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44465 (local address 127.24.20.129:55839)
0620 20:14:22.920059 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:22.920097 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:22.921275 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:22.921708 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:22.921715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:22.921797 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:22.922398 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:22.922399 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:22.922461 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:22.922551 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:22.922867 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:22.922875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:22.922880 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:14:22.923032 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 20:14:22.923312 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":112,"threads_started":1}
I20260620 20:14:22.924281 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 2009
I20260620 20:14:22.924389 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:14:22.924793 24658 external_mini_cluster.cc:1428] Setting key 06a136c82dbfca0612e49386a887b714
I20260620 20:14:22.925505 2137 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44465
I20260620 20:14:22.925721 2137 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:22.926122 2137 heartbeater.cc:511] Master 127.24.20.190:44465 requested a full tablet report, sending...
I20260620 20:14:22.926659 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44465
--builtin_ntp_servers=127.24.20.148:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 20:14:22.927731 1943 ts_manager.cc:194] Registered new tserver with Master: d84b79b6c09d4cc2a575f3a8cdde52c8 (127.24.20.129:45615)
W20260620 20:14:23.034428 2140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:23.034695 2140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:23.034765 2140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:23.038683 2140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:23.038757 2140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:23.038846 2140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:14:23.043391 2140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:23.044378 2140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:23.045782 2140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:23.056988 2145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:23.057005 2146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:23.057050 2140 server_base.cc:1034] running on GCE node
W20260620 20:14:23.057200 2148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:23.057672 2140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:23.058430 2140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:23.059628 2140 hybrid_clock.cc:648] HybridClock initialized: now 1781986463059610 us; error 23 us; skew 500 ppm
I20260620 20:14:23.061379 2140 webserver.cc:460] Webserver started at http://127.24.20.130:34365/ using document root <none> and password file <none>
I20260620 20:14:23.061959 2140 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:23.062052 2140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:23.062252 2140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:23.064200 2140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "25ba986d4fd74998a9cc78fe1cf81866"
format_stamp: "Formatted at 2026-06-20 20:14:23 on dist-test-slave-pflr"
server_key: "685c7330f87d61e06edc750da494fdef"
server_key_iv: "eca3900aa3c6c41431e95549b850b024"
server_key_version: "encryptionkey@0"
I20260620 20:14:23.064810 2140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "25ba986d4fd74998a9cc78fe1cf81866"
format_stamp: "Formatted at 2026-06-20 20:14:23 on dist-test-slave-pflr"
server_key: "685c7330f87d61e06edc750da494fdef"
server_key_iv: "eca3900aa3c6c41431e95549b850b024"
server_key_version: "encryptionkey@0"
I20260620 20:14:23.068814 2140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 20:14:23.071221 2154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:23.072436 2140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:14:23.072574 2140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "25ba986d4fd74998a9cc78fe1cf81866"
format_stamp: "Formatted at 2026-06-20 20:14:23 on dist-test-slave-pflr"
server_key: "685c7330f87d61e06edc750da494fdef"
server_key_iv: "eca3900aa3c6c41431e95549b850b024"
server_key_version: "encryptionkey@0"
I20260620 20:14:23.072733 2140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:23.077109 2140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:23.077692 2140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:23.077868 2140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:23.078415 2140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:23.079625 2140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:23.079689 2140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:23.079746 2140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:23.079782 2140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:23.091001 2140 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:37773
I20260620 20:14:23.091023 2267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:37773 every 8 connection(s)
I20260620 20:14:23.091882 2140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:14:23.092866 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 2140
I20260620 20:14:23.092983 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:14:23.093307 24658 external_mini_cluster.cc:1428] Setting key 4276591ad2574bca44f65f278ebed7c5
I20260620 20:14:23.095301 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44465
--builtin_ntp_servers=127.24.20.148:36051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 20:14:23.098816 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:23.093799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:51609 (local address 127.24.20.190:44465)
0620 20:14:23.093936 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:23.093940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:23.095213 (+ 1273us) server_negotiation.cc:407] Connection header received
0620 20:14:23.097152 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:23.097155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:23.097196 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:23.097273 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:23.098338 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:23.098341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:23.098344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:23.098356 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:23.098378 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:23.098386 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:23.098490 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:23.098643 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 20:14:23.098687 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:14:23.099622 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:23.094265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44465 (local address 127.24.20.130:51609)
0620 20:14:23.095054 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:23.095106 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:23.096951 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:23.097430 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:23.097436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:23.097519 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:23.098095 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:23.098096 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:23.098135 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:23.098198 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:23.098476 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:23.098483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:23.098488 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:14:23.098687 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 20:14:23.098906 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":152,"threads_started":1}
I20260620 20:14:23.100854 2268 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44465
I20260620 20:14:23.101081 2268 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:23.101457 2268 heartbeater.cc:511] Master 127.24.20.190:44465 requested a full tablet report, sending...
I20260620 20:14:23.102447 1943 ts_manager.cc:194] Registered new tserver with Master: 25ba986d4fd74998a9cc78fe1cf81866 (127.24.20.130:37773)
W20260620 20:14:23.203718 2271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:23.203965 2271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:23.204005 2271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:23.208117 2271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:23.208168 2271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:23.208256 2271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:14:23.212998 2271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:23.214062 2271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:23.215355 2271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:23.224812 2276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:23.224948 2279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:23.224824 2277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:23.225106 2271 server_base.cc:1034] running on GCE node
I20260620 20:14:23.225481 2271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:23.226091 2271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:23.227276 2271 hybrid_clock.cc:648] HybridClock initialized: now 1781986463227243 us; error 47 us; skew 500 ppm
I20260620 20:14:23.229153 2271 webserver.cc:460] Webserver started at http://127.24.20.131:35073/ using document root <none> and password file <none>
I20260620 20:14:23.229707 2271 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:23.229794 2271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:23.230018 2271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:23.232020 2271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "253b0fd4ede64857a65417be25187072"
format_stamp: "Formatted at 2026-06-20 20:14:23 on dist-test-slave-pflr"
server_key: "5eb73eacfba64f24295204a7024c64f0"
server_key_iv: "629c6a4d8f05ff75903c7cad35e16632"
server_key_version: "encryptionkey@0"
I20260620 20:14:23.232566 2271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "253b0fd4ede64857a65417be25187072"
format_stamp: "Formatted at 2026-06-20 20:14:23 on dist-test-slave-pflr"
server_key: "5eb73eacfba64f24295204a7024c64f0"
server_key_iv: "629c6a4d8f05ff75903c7cad35e16632"
server_key_version: "encryptionkey@0"
I20260620 20:14:23.236070 2271 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:14:23.238566 2285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:23.239749 2271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:14:23.239876 2271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "253b0fd4ede64857a65417be25187072"
format_stamp: "Formatted at 2026-06-20 20:14:23 on dist-test-slave-pflr"
server_key: "5eb73eacfba64f24295204a7024c64f0"
server_key_iv: "629c6a4d8f05ff75903c7cad35e16632"
server_key_version: "encryptionkey@0"
I20260620 20:14:23.239984 2271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:23.244488 2271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:23.245093 2271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:23.245267 2271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:23.245769 2271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:23.246604 2271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:23.246678 2271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:23.246764 2271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:23.246816 2271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:23.255646 2271 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:37077
I20260620 20:14:23.255682 2398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:37077 every 8 connection(s)
I20260620 20:14:23.256533 2271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:14:23.261550 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 2271
I20260620 20:14:23.261700 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:14:23.262033 24658 external_mini_cluster.cc:1428] Setting key 749d1486d18c650e03782e8d28664eda
I20260620 20:14:23.262728 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:23.258307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:42483 (local address 127.24.20.190:44465)
0620 20:14:23.258437 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:23.258441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:23.259461 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 20:14:23.260828 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:23.260831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:23.260900 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:23.260999 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:23.262262 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:23.262265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:23.262267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:14:23.262277 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:23.262298 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:14:23.262305 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:23.262421 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:23.262549 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 20:14:23.262595 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:14:23.263758 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:23.258706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44465 (local address 127.24.20.131:42483)
0620 20:14:23.259318 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:23.259355 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:23.260548 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:23.261159 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:23.261166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:23.261251 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:23.261956 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:14:23.261958 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:23.262011 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:14:23.262097 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:23.262414 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:23.262420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:23.262425 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:14:23.262586 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 20:14:23.262851 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260620 20:14:23.264817 2399 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44465
I20260620 20:14:23.265034 2399 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:23.265420 2399 heartbeater.cc:511] Master 127.24.20.190:44465 requested a full tablet report, sending...
I20260620 20:14:23.266342 1943 ts_manager.cc:194] Registered new tserver with Master: 253b0fd4ede64857a65417be25187072 (127.24.20.131:37077)
I20260620 20:14:23.275995 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 20:14:23.278848 1988 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:14:23.277996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.24.20.190:44465)
0620 20:14:23.278136 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:23.278141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:23.278292 (+ 151us) server_negotiation.cc:407] Connection header received
0620 20:14:23.278395 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:23.278397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:23.278425 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:23.278487 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:23.278726 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50402: BlockingRecv error: recv got EOF from 127.0.0.1:50402 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:14:23.279085 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 2009
I20260620 20:14:23.284557 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 2140
I20260620 20:14:23.289670 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 2271
I20260620 20:14:23.294492 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 1918
2026-06-20T20:14:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (788 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:23 dist-test-slave-pflr krb5kdc[2409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:23 dist-test-slave-pflr krb5kdc[2409](info): set up 2 sockets
Jun 20 20:14:23 dist-test-slave-pflr krb5kdc[2409](info): commencing operation
krb5kdc: starting...
W20260620 20:14:24.326035 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:14:24 dist-test-slave-pflr krb5kdc[2409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:14:24.440824 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:35953
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:35953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:24.548168 2422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:24.548405 2422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:24.548444 2422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:24.552531 2422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:14:24.552605 2422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:24.552626 2422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:24.552644 2422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:14:24.552706 2422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:14:24.557987 2422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:35953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:35953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:24.559181 2422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:24.560101 2422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:24.568286 2427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:24.568269 2430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:24.568261 2428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:24.568775 2422 server_base.cc:1034] running on GCE node
I20260620 20:14:24.569294 2422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:24.570346 2422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:24.571527 2422 hybrid_clock.cc:648] HybridClock initialized: now 1781986464571510 us; error 33 us; skew 500 ppm
Jun 20 20:14:24 dist-test-slave-pflr krb5kdc[2409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986464, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:24.574272 2422 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:14:24.575073 2422 webserver.cc:460] Webserver started at http://127.24.20.190:36885/ using document root <none> and password file <none>
I20260620 20:14:24.575599 2422 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:24.575692 2422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:24.575917 2422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:24.577922 2422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "d6f4e503b2db4c4dba8436798f0ba1c7"
format_stamp: "Formatted at 2026-06-20 20:14:24 on dist-test-slave-pflr"
server_key: "1b937fd427a96c025505c38186f801f9"
server_key_iv: "ca85c5ed81de09c0657c0f4a4c81ad4e"
server_key_version: "encryptionkey@0"
I20260620 20:14:24.578481 2422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "d6f4e503b2db4c4dba8436798f0ba1c7"
format_stamp: "Formatted at 2026-06-20 20:14:24 on dist-test-slave-pflr"
server_key: "1b937fd427a96c025505c38186f801f9"
server_key_iv: "ca85c5ed81de09c0657c0f4a4c81ad4e"
server_key_version: "encryptionkey@0"
I20260620 20:14:24.581975 2422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 20:14:24.584232 2437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:24.585482 2422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:14:24.585633 2422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "d6f4e503b2db4c4dba8436798f0ba1c7"
format_stamp: "Formatted at 2026-06-20 20:14:24 on dist-test-slave-pflr"
server_key: "1b937fd427a96c025505c38186f801f9"
server_key_iv: "ca85c5ed81de09c0657c0f4a4c81ad4e"
server_key_version: "encryptionkey@0"
I20260620 20:14:24.585767 2422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:24.601399 2422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:24.604194 2422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:24.604389 2422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:24.611109 2422 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:35953
I20260620 20:14:24.611147 2489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:35953 every 8 connection(s)
I20260620 20:14:24.611991 2422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:14:24.614569 2490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:24.616233 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 2422
I20260620 20:14:24.616328 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:14:24.616653 24658 external_mini_cluster.cc:1428] Setting key 31b955fe0d8346287f2fe9abacd22bd3
I20260620 20:14:24.621039 2490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7: Bootstrap starting.
I20260620 20:14:24.623484 2490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:24.624346 2490 log.cc:824] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:14:24 dist-test-slave-pflr krb5kdc[2409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:14:24.626323 2490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7: No bootstrap required, opened a new log
I20260620 20:14:24.630990 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:24.617955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.24.20.190:35953)
0620 20:14:24.618351 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:24.618369 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:24.618409 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:14:24.619204 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:24.619227 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:24.619645 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:24.619991 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:24.620666 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:24.622043 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:24.622909 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:24.623566 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:24.623749 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:24.626272 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:24.626277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:24.626291 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:24.626319 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:24.628141 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:24.628542 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:24.628549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:24.628555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:24.628740 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:24.628997 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:24.629002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:24.629005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:24.629446 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:24.629655 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:24.629827 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 20:14:24.630097 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":103,"threads_started":1}
I20260620 20:14:24.630856 2490 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6f4e503b2db4c4dba8436798f0ba1c7" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 35953 } }
I20260620 20:14:24.631083 2490 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:24.631141 2490 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6f4e503b2db4c4dba8436798f0ba1c7, State: Initialized, Role: FOLLOWER
I20260620 20:14:24.631600 2490 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6f4e503b2db4c4dba8436798f0ba1c7" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 35953 } }
I20260620 20:14:24.631747 2490 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:14:24.631810 2490 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:14:24.631901 2490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:24.632718 2490 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6f4e503b2db4c4dba8436798f0ba1c7" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 35953 } }
I20260620 20:14:24.633057 2490 leader_election.cc:302] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6f4e503b2db4c4dba8436798f0ba1c7; no voters:
I20260620 20:14:24.633358 2490 leader_election.cc:288] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:14:24.633442 2495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:24.633682 2495 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [term 1 LEADER]: Becoming Leader. State: Replica: d6f4e503b2db4c4dba8436798f0ba1c7, State: Running, Role: LEADER
I20260620 20:14:24.633966 2495 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6f4e503b2db4c4dba8436798f0ba1c7" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 35953 } }
I20260620 20:14:24.634466 2490 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:14:24.635425 2497 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6f4e503b2db4c4dba8436798f0ba1c7. Latest consensus state: current_term: 1 leader_uuid: "d6f4e503b2db4c4dba8436798f0ba1c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6f4e503b2db4c4dba8436798f0ba1c7" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 35953 } } }
I20260620 20:14:24.635547 2497 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:24.635593 2496 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6f4e503b2db4c4dba8436798f0ba1c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6f4e503b2db4c4dba8436798f0ba1c7" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 35953 } } }
I20260620 20:14:24.635684 2496 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7 [sys.catalog]: This master's current role is: LEADER
I20260620 20:14:24.636134 2504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:14:24.638796 2504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:14:24.644251 2504 catalog_manager.cc:1347] Generated new cluster ID: c3a40df156fd4c5eb8722c28105241b2
I20260620 20:14:24.644313 2504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:14:24.698429 2504 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:14:24.699725 2504 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:14:24.708468 2504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7: Generated new TSK 0
I20260620 20:14:24.709275 2504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:14:24.746033 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:35953
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:24.855692 2516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:24.855969 2516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:24.856043 2516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:24.860001 2516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:24.860090 2516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:24.860173 2516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:14:24.865384 2516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:35953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:24.866360 2516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:24.867338 2516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:24.876684 2522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:24.876648 2521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:24.876717 2524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:24.876991 2516 server_base.cc:1034] running on GCE node
I20260620 20:14:24.877398 2516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:24.878000 2516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:24.879176 2516 hybrid_clock.cc:648] HybridClock initialized: now 1781986464879145 us; error 44 us; skew 500 ppm
Jun 20 20:14:24 dist-test-slave-pflr krb5kdc[2409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986464, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:24.881763 2516 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:14:24.882525 2516 webserver.cc:460] Webserver started at http://127.24.20.129:43495/ using document root <none> and password file <none>
I20260620 20:14:24.883096 2516 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:24.883162 2516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:24.883360 2516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:24.885317 2516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "82f92139b90c40a99fc52ce2197ee108"
format_stamp: "Formatted at 2026-06-20 20:14:24 on dist-test-slave-pflr"
server_key: "b2bc494688d6d2eec2aafb022bbc1e22"
server_key_iv: "0e1284d78120334a96b38ab77b983f3c"
server_key_version: "encryptionkey@0"
I20260620 20:14:24.885843 2516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "82f92139b90c40a99fc52ce2197ee108"
format_stamp: "Formatted at 2026-06-20 20:14:24 on dist-test-slave-pflr"
server_key: "b2bc494688d6d2eec2aafb022bbc1e22"
server_key_iv: "0e1284d78120334a96b38ab77b983f3c"
server_key_version: "encryptionkey@0"
I20260620 20:14:24.889375 2516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:14:24.891713 2531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:24.892834 2516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:14:24.892942 2516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "82f92139b90c40a99fc52ce2197ee108"
format_stamp: "Formatted at 2026-06-20 20:14:24 on dist-test-slave-pflr"
server_key: "b2bc494688d6d2eec2aafb022bbc1e22"
server_key_iv: "0e1284d78120334a96b38ab77b983f3c"
server_key_version: "encryptionkey@0"
I20260620 20:14:24.893035 2516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:24.905826 2516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:24.908315 2516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:24.908499 2516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:24.909049 2516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:24.909783 2516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:24.909827 2516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:24.909924 2516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:24.909960 2516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:24.918366 2516 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:35601
I20260620 20:14:24.918404 2644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:35601 every 8 connection(s)
I20260620 20:14:24.919183 2516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:14:24.922164 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 2516
I20260620 20:14:24.922299 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:14:24.922587 24658 external_mini_cluster.cc:1428] Setting key 9896636ca2fcf8c4e880d12801963408
Jun 20 20:14:24 dist-test-slave-pflr krb5kdc[2409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986464, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:14:24.933542 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:24.920818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:47057 (local address 127.24.20.190:35953)
0620 20:14:24.920960 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:24.920965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:24.921788 (+ 823us) server_negotiation.cc:407] Connection header received
0620 20:14:24.923192 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:24.923195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:24.923248 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:24.923333 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:24.924531 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:24.925415 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:24.927136 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:24.927351 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:24.927425 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:24.930143 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:24.930145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:24.930148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:24.930170 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:24.931585 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:24.932072 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:24.932076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:24.932078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:24.932155 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:24.932474 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:24.932479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:24.932480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:24.932705 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:24.932805 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:24.933259 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 20:14:24.933376 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:14:24.934335 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:24.921226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:35953 (local address 127.24.20.129:47057)
0620 20:14:24.921650 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:24.921688 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:24.923008 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:24.923523 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:24.923531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:24.923967 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:24.924392 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:24.924404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:24.925553 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:24.925558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:24.927016 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:24.927029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:24.927491 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:24.927497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:24.927726 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:24.928370 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:24.928390 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:24.929972 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:24.931699 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:24.931704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:24.931717 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:24.931961 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:24.932242 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:24.932246 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:24.932248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:24.932379 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:24.932812 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:24.932821 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:24.933040 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 20:14:24.933219 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 20:14:24.933506 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
I20260620 20:14:24.935497 2645 heartbeater.cc:347] Connected to a master server at 127.24.20.190:35953
I20260620 20:14:24.935731 2645 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:24.936259 2645 heartbeater.cc:511] Master 127.24.20.190:35953 requested a full tablet report, sending...
I20260620 20:14:24.937687 2454 ts_manager.cc:194] Registered new tserver with Master: 82f92139b90c40a99fc52ce2197ee108 (127.24.20.129:35601)
I20260620 20:14:24.938763 2454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:47057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:14:24.949158 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:35953
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:25.055382 2650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:25.055649 2650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:25.055696 2650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:25.059672 2650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:25.059727 2650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:25.059813 2650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:14:25.064977 2650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:35953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:25.065995 2650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:25.066968 2650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:25.076296 2655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:25.076395 2650 server_base.cc:1034] running on GCE node
W20260620 20:14:25.076292 2656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:25.076356 2658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:25.076928 2650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:25.077515 2650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:25.078691 2650 hybrid_clock.cc:648] HybridClock initialized: now 1781986465078674 us; error 23 us; skew 500 ppm
Jun 20 20:14:25 dist-test-slave-pflr krb5kdc[2409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986465, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:25.082643 2650 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:14:25.083554 2650 webserver.cc:460] Webserver started at http://127.24.20.130:39207/ using document root <none> and password file <none>
I20260620 20:14:25.084232 2650 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:25.084316 2650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:25.084537 2650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:25.086845 2650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "6dc2df26ea2d4906b89321c4a7ee92e7"
format_stamp: "Formatted at 2026-06-20 20:14:25 on dist-test-slave-pflr"
server_key: "5d65c555bb8d2cd395167bae658bb9c7"
server_key_iv: "7168804f939286c41cbe8a42113fb0a6"
server_key_version: "encryptionkey@0"
I20260620 20:14:25.087340 2650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "6dc2df26ea2d4906b89321c4a7ee92e7"
format_stamp: "Formatted at 2026-06-20 20:14:25 on dist-test-slave-pflr"
server_key: "5d65c555bb8d2cd395167bae658bb9c7"
server_key_iv: "7168804f939286c41cbe8a42113fb0a6"
server_key_version: "encryptionkey@0"
I20260620 20:14:25.090816 2650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:14:25.093021 2665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:25.094132 2650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:14:25.094244 2650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "6dc2df26ea2d4906b89321c4a7ee92e7"
format_stamp: "Formatted at 2026-06-20 20:14:25 on dist-test-slave-pflr"
server_key: "5d65c555bb8d2cd395167bae658bb9c7"
server_key_iv: "7168804f939286c41cbe8a42113fb0a6"
server_key_version: "encryptionkey@0"
I20260620 20:14:25.094343 2650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:25.112195 2650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:25.115482 2650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:25.115675 2650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:25.116227 2650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:25.117069 2650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:25.117113 2650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:25.117177 2650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:25.117213 2650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:25.125717 2650 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:42587
I20260620 20:14:25.125742 2778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:42587 every 8 connection(s)
I20260620 20:14:25.126533 2650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:14:25.134814 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 2650
I20260620 20:14:25.134938 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:14:25.135293 24658 external_mini_cluster.cc:1428] Setting key 774fef7f91a706f9bf3c51844fa193ed
Jun 20 20:14:25 dist-test-slave-pflr krb5kdc[2409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986465, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:14:25.140846 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.128283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:60375 (local address 127.24.20.190:35953)
0620 20:14:25.128411 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:25.128415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:25.129300 (+ 885us) server_negotiation.cc:407] Connection header received
0620 20:14:25.130858 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:25.130862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:25.130920 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:25.131000 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:25.132089 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.132692 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.133790 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.134022 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.134131 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.137011 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:25.137014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:25.137019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:25.137046 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:25.138522 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:25.139204 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:25.139207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:25.139209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:25.139295 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:25.139682 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:25.139684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:25.139685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:25.139840 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:25.139916 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:25.140440 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 20:14:25.140580 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:14:25.141479 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.128722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:35953 (local address 127.24.20.130:60375)
0620 20:14:25.129155 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:25.129193 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:25.130648 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:25.131167 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:25.131174 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:25.131496 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:25.131942 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.131955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.132812 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.132820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.133668 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.133677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.134165 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.134169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.134330 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.135127 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:25.135150 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:25.136852 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:25.138680 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:25.138689 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:25.138706 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:25.139069 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:25.139407 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:25.139413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:25.139416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:25.139589 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:25.139938 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:25.139944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:25.140209 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 20:14:25.140358 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 20:14:25.140580 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":126,"threads_started":1}
I20260620 20:14:25.142565 2779 heartbeater.cc:347] Connected to a master server at 127.24.20.190:35953
I20260620 20:14:25.142822 2779 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:25.143378 2779 heartbeater.cc:511] Master 127.24.20.190:35953 requested a full tablet report, sending...
I20260620 20:14:25.144431 2454 ts_manager.cc:194] Registered new tserver with Master: 6dc2df26ea2d4906b89321c4a7ee92e7 (127.24.20.130:42587)
I20260620 20:14:25.145268 2454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:60375
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:14:25.161880 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:35953
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:14:25.270390 2784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:14:25.270617 2784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:14:25.270658 2784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:14:25.274773 2784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:14:25.274820 2784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:14:25.274875 2784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:14:25.279873 2784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:35953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:14:25.280835 2784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:14:25.281796 2784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:14:25.291062 2789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:25.291083 2790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:14:25.291214 2792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:14:25.291314 2784 server_base.cc:1034] running on GCE node
I20260620 20:14:25.291745 2784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:14:25.292337 2784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:14:25.293493 2784 hybrid_clock.cc:648] HybridClock initialized: now 1781986465293472 us; error 29 us; skew 500 ppm
Jun 20 20:14:25 dist-test-slave-pflr krb5kdc[2409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986465, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:14:25.296082 2784 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:14:25.297086 2784 webserver.cc:460] Webserver started at http://127.24.20.131:45595/ using document root <none> and password file <none>
I20260620 20:14:25.297696 2784 fs_manager.cc:359] Metadata directory not provided
I20260620 20:14:25.297782 2784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:14:25.298069 2784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:14:25.300099 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "e8dfa42bd6fd4875970f7969e7689760"
format_stamp: "Formatted at 2026-06-20 20:14:25 on dist-test-slave-pflr"
server_key: "43ab0064ca600cc4d40dc05e06507545"
server_key_iv: "f68b7ed08d238dab9c831f3bfe9f5663"
server_key_version: "encryptionkey@0"
I20260620 20:14:25.300726 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "e8dfa42bd6fd4875970f7969e7689760"
format_stamp: "Formatted at 2026-06-20 20:14:25 on dist-test-slave-pflr"
server_key: "43ab0064ca600cc4d40dc05e06507545"
server_key_iv: "f68b7ed08d238dab9c831f3bfe9f5663"
server_key_version: "encryptionkey@0"
I20260620 20:14:25.304222 2784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:14:25.306569 2799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:25.307847 2784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:14:25.307998 2784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "e8dfa42bd6fd4875970f7969e7689760"
format_stamp: "Formatted at 2026-06-20 20:14:25 on dist-test-slave-pflr"
server_key: "43ab0064ca600cc4d40dc05e06507545"
server_key_iv: "f68b7ed08d238dab9c831f3bfe9f5663"
server_key_version: "encryptionkey@0"
I20260620 20:14:25.308110 2784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:14:25.323614 2784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:14:25.326371 2784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:14:25.326579 2784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:14:25.327140 2784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:14:25.327982 2784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:14:25.328054 2784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:25.328118 2784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:14:25.328168 2784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:14:25.336970 2784 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:41587
I20260620 20:14:25.337034 2912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:41587 every 8 connection(s)
I20260620 20:14:25.337844 2784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
Jun 20 20:14:25 dist-test-slave-pflr krb5kdc[2409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986465, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:14:25.348035 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 2784
I20260620 20:14:25.348167 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:14:25.348527 24658 external_mini_cluster.cc:1428] Setting key 69812a4ee04a26eefe27ea742c7a5f6f
I20260620 20:14:25.351261 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.339515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:48527 (local address 127.24.20.190:35953)
0620 20:14:25.339644 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:25.339648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:25.340417 (+ 769us) server_negotiation.cc:407] Connection header received
0620 20:14:25.341838 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:25.341841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:25.341899 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:25.342006 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:25.343084 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.343610 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.344751 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.344966 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.345079 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.347396 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:25.347399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:25.347401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:25.347424 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:25.349065 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:25.349642 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:25.349645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:25.349646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:25.349753 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:25.350156 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:25.350160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:25.350162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:25.350376 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:25.350479 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:25.351039 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 20:14:25.351149 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:14:25.352034 2915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.339871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:35953 (local address 127.24.20.131:48527)
0620 20:14:25.340287 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:25.340325 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:25.341632 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:25.342171 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:25.342178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:25.342510 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:14:25.342943 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.342955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.343730 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.343734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.344570 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.344578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.345110 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.345115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.345283 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.345850 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:14:25.345866 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:14:25.347225 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:14:25.349193 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:25.349199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:25.349212 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:25.349534 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:25.349877 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:14:25.349882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:14:25.349884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:14:25.350060 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:14:25.350476 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:14:25.350481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:14:25.350798 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 20:14:25.350982 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 20:14:25.351218 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":109,"threads_started":1}
I20260620 20:14:25.353068 2913 heartbeater.cc:347] Connected to a master server at 127.24.20.190:35953
I20260620 20:14:25.353305 2913 heartbeater.cc:464] Registering TS with master...
I20260620 20:14:25.353762 2913 heartbeater.cc:511] Master 127.24.20.190:35953 requested a full tablet report, sending...
I20260620 20:14:25.354694 2454 ts_manager.cc:194] Registered new tserver with Master: e8dfa42bd6fd4875970f7969e7689760 (127.24.20.131:41587)
I20260620 20:14:25.355216 2454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:48527
I20260620 20:14:25.362284 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:14:25.370954 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.364098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.24.20.190:35953)
0620 20:14:25.364190 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:25.364194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:25.364449 (+ 255us) server_negotiation.cc:407] Connection header received
0620 20:14:25.364532 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:25.364534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:25.364569 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:25.364682 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:14:25.365329 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.365818 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.366607 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.366803 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.366875 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.368214 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:14:25.368217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:14:25.368219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:14:25.368235 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:14:25.369485 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:25.369977 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:25.369986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:25.369988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:25.370059 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:14:25.370283 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:14:25.370286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:14:25.370287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:14:25.370475 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:14:25.370575 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:25.370757 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 20:14:25.370843 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 20:14:25.374027 2454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:14:25.377056 2454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:14:25.391193 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.385037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.24.20.131:41587)
0620 20:14:25.385968 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:25.385977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:25.385994 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:14:25.386050 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:25.386054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:25.386260 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:25.386408 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:25.386878 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.388084 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.389582 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.390400 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.390510 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.390642 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:25.390951 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 20:14:25.391039 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":138,"threads_started":1}
I20260620 20:14:25.391204 2925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.385293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:42587 (local address 127.0.0.1:56794)
0620 20:14:25.385791 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:25.385805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:25.385900 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:25.386731 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:25.386734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:25.386747 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:25.386845 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.386850 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.388221 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.388225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.389556 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.389562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.390554 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.390557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.390718 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.390799 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 20:14:25.390907 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 20:14:25.391045 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":167,"thread_start_us":83,"threads_started":1}
I20260620 20:14:25.391204 2924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.384928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:41587 (local address 127.0.0.1:45040)
0620 20:14:25.385506 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:25.385550 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:25.385673 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:25.386591 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:25.386596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:25.386621 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:25.386750 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.386756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.388255 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.388258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.389473 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.389481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.390562 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.390565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.390732 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.390801 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 20:14:25.390907 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:14:25.391045 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":190,"thread_start_us":81,"threads_started":1}
I20260620 20:14:25.392319 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.385043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:35601 (local address 127.0.0.1:34538)
0620 20:14:25.385710 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:25.385724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:25.385796 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:25.386919 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:25.386922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:25.386938 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:25.387038 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.387047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.387963 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.387966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.389551 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.389561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.391940 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.391944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.392028 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.392049 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:14:25.392119 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:14:25.392187 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":97,"thread_start_us":95,"threads_started":1}
I20260620 20:14:25.392495 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.385718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.24.20.130:42587)
0620 20:14:25.386301 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:25.386312 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:25.386334 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:14:25.386409 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:25.386415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:25.386598 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:25.386739 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:25.386975 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.388084 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.389651 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.390431 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.390520 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.390642 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:25.391601 (+ 959us) server_negotiation.cc:299] Negotiation successful
0620 20:14:25.392340 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":126,"threads_started":1}
I20260620 20:14:25.392750 2929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.386103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.24.20.129:35601)
0620 20:14:25.386522 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:25.386532 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:25.386551 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:14:25.386618 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:25.386623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:25.386790 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:25.386920 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:25.387136 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.387829 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.390701 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.391628 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.391756 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.391882 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:25.392477 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 20:14:25.392574 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":135,"threads_started":1}
I20260620 20:14:25.394434 2847 tablet_service.cc:1467] Processing CreateTablet for tablet 39600248cf7f47c2b8ab270ee92969f1 (DEFAULT_TABLE table=test-table [id=b0279d66884c4629b47360dfa9da239b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:25.394701 2713 tablet_service.cc:1467] Processing CreateTablet for tablet 39600248cf7f47c2b8ab270ee92969f1 (DEFAULT_TABLE table=test-table [id=b0279d66884c4629b47360dfa9da239b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:25.395484 2713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39600248cf7f47c2b8ab270ee92969f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:25.395526 2847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39600248cf7f47c2b8ab270ee92969f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:25.395655 2579 tablet_service.cc:1467] Processing CreateTablet for tablet 39600248cf7f47c2b8ab270ee92969f1 (DEFAULT_TABLE table=test-table [id=b0279d66884c4629b47360dfa9da239b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:14:25.396546 2579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39600248cf7f47c2b8ab270ee92969f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:14:25.402256 2930 tablet_bootstrap.cc:492] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7: Bootstrap starting.
I20260620 20:14:25.402841 2931 tablet_bootstrap.cc:492] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760: Bootstrap starting.
I20260620 20:14:25.405011 2932 tablet_bootstrap.cc:492] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108: Bootstrap starting.
I20260620 20:14:25.405303 2931 tablet_bootstrap.cc:654] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:25.405303 2930 tablet_bootstrap.cc:654] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:25.406164 2931 log.cc:824] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:25.406164 2930 log.cc:824] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:25.406833 2932 tablet_bootstrap.cc:654] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108: Neither blocks nor log segments found. Creating new log.
I20260620 20:14:25.407958 2932 log.cc:824] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108: Log is configured to *not* fsync() on all Append() calls
I20260620 20:14:25.408174 2931 tablet_bootstrap.cc:492] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760: No bootstrap required, opened a new log
I20260620 20:14:25.408174 2930 tablet_bootstrap.cc:492] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7: No bootstrap required, opened a new log
I20260620 20:14:25.408356 2931 ts_tablet_manager.cc:1397] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:14:25.408356 2930 ts_tablet_manager.cc:1397] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:14:25.409782 2932 tablet_bootstrap.cc:492] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108: No bootstrap required, opened a new log
I20260620 20:14:25.409976 2932 ts_tablet_manager.cc:1397] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:14:25.412132 2930 raft_consensus.cc:348] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dfa42bd6fd4875970f7969e7689760" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41587 } } peers { permanent_uuid: "82f92139b90c40a99fc52ce2197ee108" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 35601 } } peers { permanent_uuid: "6dc2df26ea2d4906b89321c4a7ee92e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42587 } }
I20260620 20:14:25.412123 2931 raft_consensus.cc:348] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dfa42bd6fd4875970f7969e7689760" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41587 } } peers { permanent_uuid: "82f92139b90c40a99fc52ce2197ee108" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 35601 } } peers { permanent_uuid: "6dc2df26ea2d4906b89321c4a7ee92e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42587 } }
I20260620 20:14:25.412377 2931 raft_consensus.cc:374] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:25.412377 2930 raft_consensus.cc:374] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:25.412431 2930 raft_consensus.cc:729] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc2df26ea2d4906b89321c4a7ee92e7, State: Initialized, Role: FOLLOWER
I20260620 20:14:25.412431 2931 raft_consensus.cc:729] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8dfa42bd6fd4875970f7969e7689760, State: Initialized, Role: FOLLOWER
I20260620 20:14:25.412912 2930 consensus_queue.cc:260] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dfa42bd6fd4875970f7969e7689760" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41587 } } peers { permanent_uuid: "82f92139b90c40a99fc52ce2197ee108" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 35601 } } peers { permanent_uuid: "6dc2df26ea2d4906b89321c4a7ee92e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42587 } }
I20260620 20:14:25.412912 2931 consensus_queue.cc:260] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dfa42bd6fd4875970f7969e7689760" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41587 } } peers { permanent_uuid: "82f92139b90c40a99fc52ce2197ee108" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 35601 } } peers { permanent_uuid: "6dc2df26ea2d4906b89321c4a7ee92e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42587 } }
I20260620 20:14:25.413084 2932 raft_consensus.cc:348] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dfa42bd6fd4875970f7969e7689760" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41587 } } peers { permanent_uuid: "82f92139b90c40a99fc52ce2197ee108" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 35601 } } peers { permanent_uuid: "6dc2df26ea2d4906b89321c4a7ee92e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42587 } }
I20260620 20:14:25.413327 2932 raft_consensus.cc:374] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:14:25.413391 2932 raft_consensus.cc:729] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f92139b90c40a99fc52ce2197ee108, State: Initialized, Role: FOLLOWER
I20260620 20:14:25.413849 2930 ts_tablet_manager.cc:1428] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:14:25.413849 2931 ts_tablet_manager.cc:1428] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:14:25.413820 2932 consensus_queue.cc:260] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dfa42bd6fd4875970f7969e7689760" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41587 } } peers { permanent_uuid: "82f92139b90c40a99fc52ce2197ee108" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 35601 } } peers { permanent_uuid: "6dc2df26ea2d4906b89321c4a7ee92e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42587 } }
I20260620 20:14:25.414420 2913 heartbeater.cc:503] Master 127.24.20.190:35953 was elected leader, sending a full tablet report...
I20260620 20:14:25.414527 2779 heartbeater.cc:503] Master 127.24.20.190:35953 was elected leader, sending a full tablet report...
I20260620 20:14:25.414522 2645 heartbeater.cc:503] Master 127.24.20.190:35953 was elected leader, sending a full tablet report...
I20260620 20:14:25.415627 2932 ts_tablet_manager.cc:1428] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 20:14:25.420173 2646 tablet.cc:2369] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:14:25.588973 2914 tablet.cc:2369] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:14:25.623570 2938 raft_consensus.cc:482] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:14:25.623790 2938 raft_consensus.cc:504] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dfa42bd6fd4875970f7969e7689760" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41587 } } peers { permanent_uuid: "82f92139b90c40a99fc52ce2197ee108" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 35601 } } peers { permanent_uuid: "6dc2df26ea2d4906b89321c4a7ee92e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42587 } }
I20260620 20:14:25.624941 2938 leader_election.cc:288] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8dfa42bd6fd4875970f7969e7689760 (127.24.20.131:41587), 6dc2df26ea2d4906b89321c4a7ee92e7 (127.24.20.130:42587)
W20260620 20:14:25.627559 2780 tablet.cc:2369] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:14:25.628952 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.625133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:41587 (local address 127.24.20.129:41289)
0620 20:14:25.625491 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:25.625507 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:25.625652 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:25.625908 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:25.625911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:25.625926 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:25.626043 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.626049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.626765 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.626771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.627691 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.627701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.628532 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.628535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.628663 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.628684 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:14:25.628746 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 20:14:25.628802 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":4,"thread_start_us":96,"threads_started":1}
I20260620 20:14:25.628986 2926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.625236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:41289 (local address 127.24.20.131:41587)
0620 20:14:25.625379 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:25.625383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:25.625595 (+ 212us) server_negotiation.cc:407] Connection header received
0620 20:14:25.625765 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:25.625768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:25.625811 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:25.625887 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:25.626156 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.626659 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.627840 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.628426 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.628531 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.628618 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:25.628787 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 20:14:25.628861 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:14:25.629133 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.625211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:42587 (local address 127.24.20.129:33073)
0620 20:14:25.625560 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:25.625572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:14:25.625693 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:14:25.625959 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:14:25.625962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:14:25.625976 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:14:25.626118 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.626124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.626820 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.626826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.627621 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:14:25.627653 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.628658 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.628661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:14:25.628799 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.628819 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:14:25.628894 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 20:14:25.628954 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":9,"thread_start_us":105,"threads_started":1}
I20260620 20:14:25.629151 2928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.625313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:33073 (local address 127.24.20.130:42587)
0620 20:14:25.625441 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:25.625446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:25.625661 (+ 215us) server_negotiation.cc:407] Connection header received
0620 20:14:25.625820 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:25.625823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:25.625867 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:25.625953 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:14:25.626227 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.626710 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.627783 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.628387 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.628492 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.628547 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:25.628929 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 20:14:25.629012 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:14:25.629590 2867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39600248cf7f47c2b8ab270ee92969f1" candidate_uuid: "82f92139b90c40a99fc52ce2197ee108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8dfa42bd6fd4875970f7969e7689760" is_pre_election: true
I20260620 20:14:25.629781 2733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39600248cf7f47c2b8ab270ee92969f1" candidate_uuid: "82f92139b90c40a99fc52ce2197ee108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc2df26ea2d4906b89321c4a7ee92e7" is_pre_election: true
I20260620 20:14:25.629886 2867 raft_consensus.cc:2454] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f92139b90c40a99fc52ce2197ee108 in term 0.
I20260620 20:14:25.630072 2733 raft_consensus.cc:2454] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f92139b90c40a99fc52ce2197ee108 in term 0.
I20260620 20:14:25.630379 2534 leader_election.cc:302] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f92139b90c40a99fc52ce2197ee108, e8dfa42bd6fd4875970f7969e7689760; no voters:
I20260620 20:14:25.630657 2938 raft_consensus.cc:2781] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:14:25.630738 2938 raft_consensus.cc:482] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:14:25.630781 2938 raft_consensus.cc:3037] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:25.631745 2938 raft_consensus.cc:504] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dfa42bd6fd4875970f7969e7689760" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41587 } } peers { permanent_uuid: "82f92139b90c40a99fc52ce2197ee108" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 35601 } } peers { permanent_uuid: "6dc2df26ea2d4906b89321c4a7ee92e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42587 } }
I20260620 20:14:25.632068 2938 leader_election.cc:288] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [CANDIDATE]: Term 1 election: Requested vote from peers e8dfa42bd6fd4875970f7969e7689760 (127.24.20.131:41587), 6dc2df26ea2d4906b89321c4a7ee92e7 (127.24.20.130:42587)
I20260620 20:14:25.632344 2867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39600248cf7f47c2b8ab270ee92969f1" candidate_uuid: "82f92139b90c40a99fc52ce2197ee108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8dfa42bd6fd4875970f7969e7689760"
I20260620 20:14:25.632376 2733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39600248cf7f47c2b8ab270ee92969f1" candidate_uuid: "82f92139b90c40a99fc52ce2197ee108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dc2df26ea2d4906b89321c4a7ee92e7"
I20260620 20:14:25.632477 2867 raft_consensus.cc:3037] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:25.632469 2733 raft_consensus.cc:3037] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:14:25.633472 2733 raft_consensus.cc:2454] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f92139b90c40a99fc52ce2197ee108 in term 1.
I20260620 20:14:25.633476 2867 raft_consensus.cc:2454] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f92139b90c40a99fc52ce2197ee108 in term 1.
I20260620 20:14:25.633734 2533 leader_election.cc:302] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dc2df26ea2d4906b89321c4a7ee92e7, 82f92139b90c40a99fc52ce2197ee108; no voters:
I20260620 20:14:25.633915 2938 raft_consensus.cc:2781] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:14:25.634137 2938 raft_consensus.cc:686] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [term 1 LEADER]: Becoming Leader. State: Replica: 82f92139b90c40a99fc52ce2197ee108, State: Running, Role: LEADER
I20260620 20:14:25.634449 2938 consensus_queue.cc:237] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dfa42bd6fd4875970f7969e7689760" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41587 } } peers { permanent_uuid: "82f92139b90c40a99fc52ce2197ee108" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 35601 } } peers { permanent_uuid: "6dc2df26ea2d4906b89321c4a7ee92e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42587 } }
I20260620 20:14:25.637581 2452 catalog_manager.cc:5495] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82f92139b90c40a99fc52ce2197ee108 (127.24.20.129). New cstate: current_term: 1 leader_uuid: "82f92139b90c40a99fc52ce2197ee108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8dfa42bd6fd4875970f7969e7689760" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f92139b90c40a99fc52ce2197ee108" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 35601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dc2df26ea2d4906b89321c4a7ee92e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20260620 20:14:25.680968 2929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.677620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.24.20.129:35601)
0620 20:14:25.677774 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 20:14:25.677778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:14:25.677792 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:14:25.677890 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:14:25.677892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:14:25.677950 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:14:25.678056 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:14:25.678261 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.678797 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.679577 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:14:25.679794 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:14:25.679874 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:14:25.680122 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:14:25.680537 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:14:25.680655 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:14:25.680731 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 20:14:25.680778 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:14:25.686568 2733 raft_consensus.cc:1261] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7 [term 1 FOLLOWER]: Refusing update from remote peer 82f92139b90c40a99fc52ce2197ee108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:25.686590 2867 raft_consensus.cc:1261] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760 [term 1 FOLLOWER]: Refusing update from remote peer 82f92139b90c40a99fc52ce2197ee108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:14:25.687228 2938 consensus_queue.cc:1035] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc2df26ea2d4906b89321c4a7ee92e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:25.687448 2941 consensus_queue.cc:1035] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8dfa42bd6fd4875970f7969e7689760" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:14:25.703071 2452 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:25.703254 2452 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:14:25.705621 2452 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6f4e503b2db4c4dba8436798f0ba1c7: Sending DeleteTablet for 3 replicas of tablet 39600248cf7f47c2b8ab270ee92969f1
I20260620 20:14:25.706313 2579 tablet_service.cc:1514] Processing DeleteTablet for tablet 39600248cf7f47c2b8ab270ee92969f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:25 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:34538
I20260620 20:14:25.706385 2847 tablet_service.cc:1514] Processing DeleteTablet for tablet 39600248cf7f47c2b8ab270ee92969f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:25 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:45040
I20260620 20:14:25.706770 2954 tablet_replica.cc:330] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108: stopping tablet replica
I20260620 20:14:25.707063 2954 raft_consensus.cc:2229] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:14:25.707135 2713 tablet_service.cc:1514] Processing DeleteTablet for tablet 39600248cf7f47c2b8ab270ee92969f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:14:25 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:56794
I20260620 20:14:25.707314 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 2516
I20260620 20:14:25.707381 2954 raft_consensus.cc:2258] T 39600248cf7f47c2b8ab270ee92969f1 P 82f92139b90c40a99fc52ce2197ee108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:25.708967 2956 tablet_replica.cc:330] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7: stopping tablet replica
I20260620 20:14:25.709160 2956 raft_consensus.cc:2229] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:25.709426 2956 raft_consensus.cc:2258] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:25.709766 2955 tablet_replica.cc:330] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760: stopping tablet replica
I20260620 20:14:25.710005 2955 raft_consensus.cc:2229] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:14:25.710261 2955 raft_consensus.cc:2258] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:14:25.710942 2956 ts_tablet_manager.cc:1905] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:25.711525 2955 ts_tablet_manager.cc:1905] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:14:25.713922 2955 ts_tablet_manager.cc:1918] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:25.714023 2955 log.cc:1196] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/39600248cf7f47c2b8ab270ee92969f1
I20260620 20:14:25.714325 2955 ts_tablet_manager.cc:1939] T 39600248cf7f47c2b8ab270ee92969f1 P e8dfa42bd6fd4875970f7969e7689760: Deleting consensus metadata
I20260620 20:14:25.714322 2956 ts_tablet_manager.cc:1918] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:14:25.714409 2956 log.cc:1196] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/39600248cf7f47c2b8ab270ee92969f1
W20260620 20:14:25.714610 2438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:35601 (error 108)
I20260620 20:14:25.714733 2956 ts_tablet_manager.cc:1939] T 39600248cf7f47c2b8ab270ee92969f1 P 6dc2df26ea2d4906b89321c4a7ee92e7: Deleting consensus metadata
I20260620 20:14:25.715368 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 2650
I20260620 20:14:25.715508 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.715254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:35601 (local address 127.0.0.1:34554)
0620 20:14:25.715358 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:25.715423 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:35601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 20:14:25.715660 2438 catalog_manager.cc:4584] TS 82f92139b90c40a99fc52ce2197ee108 (127.24.20.129:35601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 39600248cf7f47c2b8ab270ee92969f1: Network error: Client connection negotiation failed: client connection to 127.24.20.129:35601: connect: Connection refused (error 111)
I20260620 20:14:25.715975 2440 catalog_manager.cc:4841] TS e8dfa42bd6fd4875970f7969e7689760 (127.24.20.131:41587): tablet 39600248cf7f47c2b8ab270ee92969f1 (table test-table [id=b0279d66884c4629b47360dfa9da239b]) successfully deleted
I20260620 20:14:25.720745 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 2784
I20260620 20:14:25.721515 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:14:25.720874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:42587 (local address 127.0.0.1:56798)
0620 20:14:25.721372 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 20:14:25.721432 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.130:42587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407}
I20260620 20:14:25.726089 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 2422
2026-06-20T20:14:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2432 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:14:25 dist-test-slave-pflr krb5kdc[2960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:14:25 dist-test-slave-pflr krb5kdc[2960](info): set up 2 sockets
Jun 20 20:14:25 dist-test-slave-pflr krb5kdc[2960](info): commencing operation
krb5kdc: starting...
W20260620 20:14:26.766098 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:14:26 dist-test-slave-pflr krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:14:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.20.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.20.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/rangeradmin_127.24.20.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/rangeradmin_127.24.20.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.20.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.20.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/rangerlookup_127.24.20.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/rangerlookup_127.24.20.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.20.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.20.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/HTTP_127.24.20.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/HTTP_127.24.20.148@KRBTEST.COM.keytab.
I20260620 20:14:26.929358 24658 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVWkq7u/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/postgres -l logfile start
2026-06-20 20:14:29.380 UTC [2990] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 20:14:29.380 UTC [2990] LOG: listening on IPv4 address "127.24.20.148", port 48819
2026-06-20 20:14:29.383 UTC [2990] LOG: listening on Unix socket "/tmp/.s.PGSQL.48819"
2026-06-20 20:14:29.403 UTC [2993] LOG: database system was shut down at 2026-06-20 20:14:27 UTC
2026-06-20 20:14:29.407 UTC [2990] LOG: database system is ready to accept connections
I20260620 20:14:30.353847 24658 mini_postgres.cc:96] Postgres bound to 48819
2026-06-20 20:14:30.360 UTC [3001] FATAL: database "slave" does not exist
127.24.20.148:48819 - accepting connections
I20260620 20:14:30.360782 24658 mini_ranger.cc:161] Starting Ranger...
I20260620 20:14:30.376070 24658 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 20:14:30.855365 24658 mini_ranger.cc:87] Created ranger Postgres database
I20260620 20:14:30.855464 24658 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-admin
2026-06-20 20:14:31,118 [I] DB FLAVOR :POSTGRES
2026-06-20 20:14:31,118 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:14:31,118 [I] Checking connection..
2026-06-20 20:14:31,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 20:14:31,350 [I] Checking connection passed.
2026-06-20 20:14:31,351 [I] --------- Verifying version history table ---------
2026-06-20 20:14:31,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:14:31,576 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:14:31,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:14:31,831 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:14:31,831 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 20:14:31,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 20:14:32.069 UTC [3065] WARNING: there is no transaction in progress
2026-06-20 20:14:32,075 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 20:14:32,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:14:32,322 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 20:14:32,323 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:14:32,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 20:14:32,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 20:14:32,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pflr.c.gcp-upstream.internal','N') ;"
2026-06-20 20:14:33,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:14:33,297 [I] Table x_portal_user does not exist in database ranger
2026-06-20 20:14:33,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:14:33,542 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 20:14:33,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:14:33,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:14:34,024 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 20:14:34,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 20:14:34.946 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:34.963 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:34.979 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:34.995 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:35.011 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:35.028 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:35.087 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:35.094 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:35.108 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:35.118 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:35.128 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:35.140 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:35.150 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:35.156 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:35.163 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:35.170 UTC [3173] WARNING: there is no transaction in progress
2026-06-20 20:14:35,794 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 20:14:35,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 20:14:36,037 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 20:14:36,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:14:36,262 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 20:14:36,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:14:36,494 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 20:14:36,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:14:36,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:14:37,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pflr.c.gcp-upstream.internal';"
2026-06-20 20:14:37,231 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 20:14:37,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 20:14:37,476 [I] DB_PATCHES have already been applied
I20260620 20:14:37.482196 24658 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVWkq7u/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 20:14:37.482282 24658 mini_ranger.cc:193] Using host: 127.24.20.148
I20260620 20:14:37.485764 24658 mini_ranger.cc:227] Ranger admin URL: http://127.24.20.148:41759
Jun 20, 2026 8:14:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:14:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:14:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:14:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:14:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:14:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:14:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 20:14:38.608975 24658 mini_ranger.cc:160] Time spent starting Ranger: real 8.248s user 0.005s sys 0.006s
Jun 20, 2026 8:14:57 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:14:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Jun 20, 2026 8:14:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:14:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:14:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:14:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:14:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:14:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:14:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:14:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:14:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:14:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:14:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:14:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:14:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:14:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:14:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:14:58 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 20:14:58.618443 24658 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.20.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.20.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/rangerkms_127.24.20.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/rangerkms_127.24.20.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.20.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.20.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.20.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/HTTP_127.24.20.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.20.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/HTTP_127.24.20.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 20:14:58 dist-test-slave-pflr krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986498, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 20:14:58.711719 24658 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 20:14:58.721318 24658 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 20:14:59.577201 24658 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 20:14:59.577309 24658 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-kms
I20260620 20:14:59.577417 24658 mini_ranger_kms.cc:227] Using postgres at 127.24.20.148:48819
2026-06-20 20:14:59,770 [I] DB FLAVOR :POSTGRES
2026-06-20 20:14:59,770 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:14:59,770 [I] Checking connection
2026-06-20 20:14:59,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 20:15:00,001 [I] connection success
2026-06-20 20:15:00,001 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 20:15:00,001 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 20:15:00,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 20:15:00,255 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 20:15:00,256 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:15:00,256 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 20:15:00,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:48819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 20:15:00,506 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 20:15:00.970902 24658 mini_ranger_kms.cc:310] Created kms service
I20260620 20:15:01.186270 24658 mini_ranger_kms.cc:326] Created kudu user
I20260620 20:15:01.231971 24658 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 20:15:01 dist-test-slave-pflr krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986498, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:01.636672 24658 mini_ranger_kms.cc:384] Added ranger policy
I20260620 20:15:01.636831 24658 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 20:15:01.636857 24658 mini_ranger_kms.cc:240] Using host: 127.24.20.148
I20260620 20:15:01.641544 24658 mini_ranger_kms.cc:276] Ranger KMS PID: 3353
I20260620 20:15:01.641672 24658 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.20.148:35553
Jun 20, 2026 8:15:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:15:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:15:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:15:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:15:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:15:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 20:15:03 dist-test-slave-pflr krb5kdc[2960](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986503, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.20.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:03.886289 24658 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.175s user 0.022s sys 0.154s
I20260620 20:15:03.886435 24658 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 20:15:03.886556 24658 mini_ranger_kms.cc:401] 127.24.20.148:35553/kms/v1/keys
Jun 20 20:15:04 dist-test-slave-pflr krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986503, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.20.148@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
Jun 20 20:15:04 dist-test-slave-pflr krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986503, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.20.148@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
Jun 20 20:15:06 dist-test-slave-pflr krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:15:06.609746 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:46011
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:46011
--ranger_config_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.20.148:35553/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:06.719712 3404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:06.719942 3404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:06.720007 3404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:06.723908 3404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:06.723963 3404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:06.723984 3404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:06.723999 3404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:06.724058 3404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:06.729305 3404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.20.148:35553/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:46011
--ranger_config_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:46011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:06.730275 3404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:06.731235 3404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:15:06.740173 3404 server_base.cc:1034] running on GCE node
W20260620 20:15:06.740154 3409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:06.740149 3410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:06.740319 3412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:06.740970 3404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:06.741941 3404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:06.743095 3404 hybrid_clock.cc:648] HybridClock initialized: now 1781986506743075 us; error 32 us; skew 500 ppm
Jun 20 20:15:06 dist-test-slave-pflr krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986506, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:06.745817 3404 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:15:06.746652 3404 webserver.cc:460] Webserver started at http://127.24.20.190:40659/ using document root <none> and password file <none>
I20260620 20:15:06.747169 3404 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:06.747222 3404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:06.747380 3404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:15:06 dist-test-slave-pflr krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986506, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:06.794917 3404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "e14476a1a3534b08b5a85d44bd8af503"
format_stamp: "Formatted at 2026-06-20 20:15:06 on dist-test-slave-pflr"
server_key: "190ccaa05a0adc218449100616b82b80"
server_key_iv: "974df2eba38991c7d1a4fdfee5c64869"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:06.795570 3404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "e14476a1a3534b08b5a85d44bd8af503"
format_stamp: "Formatted at 2026-06-20 20:15:06 on dist-test-slave-pflr"
server_key: "190ccaa05a0adc218449100616b82b80"
server_key_iv: "974df2eba38991c7d1a4fdfee5c64869"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:06.799707 3404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 20:15:06.813833 3419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:06.815326 3404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:15:06.815539 3404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "e14476a1a3534b08b5a85d44bd8af503"
format_stamp: "Formatted at 2026-06-20 20:15:06 on dist-test-slave-pflr"
server_key: "190ccaa05a0adc218449100616b82b80"
server_key_iv: "974df2eba38991c7d1a4fdfee5c64869"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:06.815689 3404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:06.827229 3404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:06.830200 3404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:06.830400 3404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:06.838152 3404 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:46011
I20260620 20:15:06.838172 3471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:46011 every 8 connection(s)
I20260620 20:15:06.839098 3404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:06.841931 3472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:06.847049 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 3404
I20260620 20:15:06.847183 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:15:06.847486 24658 external_mini_cluster.cc:1428] Setting key 3326e08a7020f60bae633a2c3c9201aa
I20260620 20:15:06.847584 3472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503: Bootstrap starting.
I20260620 20:15:06.850260 3472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:06.851100 3472 log.cc:824] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:06.853096 3472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503: No bootstrap required, opened a new log
Jun 20 20:15:06 dist-test-slave-pflr krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:15:06.857338 3472 raft_consensus.cc:348] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14476a1a3534b08b5a85d44bd8af503" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 46011 } }
I20260620 20:15:06.857581 3472 raft_consensus.cc:374] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:06.857645 3472 raft_consensus.cc:729] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14476a1a3534b08b5a85d44bd8af503, State: Initialized, Role: FOLLOWER
I20260620 20:15:06.858076 3472 consensus_queue.cc:260] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14476a1a3534b08b5a85d44bd8af503" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 46011 } }
I20260620 20:15:06.858259 3472 raft_consensus.cc:388] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:06.858333 3472 raft_consensus.cc:482] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:06.858469 3472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:06.859282 3472 raft_consensus.cc:504] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14476a1a3534b08b5a85d44bd8af503" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 46011 } }
I20260620 20:15:06.859609 3472 leader_election.cc:302] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e14476a1a3534b08b5a85d44bd8af503; no voters:
I20260620 20:15:06.859931 3472 leader_election.cc:288] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:15:06.860088 3477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:06.860301 3477 raft_consensus.cc:686] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [term 1 LEADER]: Becoming Leader. State: Replica: e14476a1a3534b08b5a85d44bd8af503, State: Running, Role: LEADER
I20260620 20:15:06.860711 3477 consensus_queue.cc:237] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14476a1a3534b08b5a85d44bd8af503" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 46011 } }
I20260620 20:15:06.861265 3472 sys_catalog.cc:564] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:06.862377 3479 sys_catalog.cc:455] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e14476a1a3534b08b5a85d44bd8af503. Latest consensus state: current_term: 1 leader_uuid: "e14476a1a3534b08b5a85d44bd8af503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14476a1a3534b08b5a85d44bd8af503" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 46011 } } }
I20260620 20:15:06.862514 3479 sys_catalog.cc:458] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:06.862699 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:06.848860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38566 (local address 127.24.20.190:46011)
0620 20:15:06.849254 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:06.849272 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:06.849313 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:15:06.850088 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:06.850111 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:06.850546 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:06.850906 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:06.851589 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:06.852702 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:06.853849 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:06.854299 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:06.854413 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:06.857261 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:06.857266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:06.857278 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:06.857305 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:06.859085 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:06.859570 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:06.859576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:06.859582 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:06.859700 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:06.860029 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:06.860032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:06.860034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:06.860552 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:06.860785 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:06.861352 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 20:15:06.861697 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
I20260620 20:15:06.862836 3478 sys_catalog.cc:455] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e14476a1a3534b08b5a85d44bd8af503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e14476a1a3534b08b5a85d44bd8af503" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 46011 } } }
I20260620 20:15:06.862965 3478 sys_catalog.cc:458] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:06.863389 3472 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:15:06.863965 3486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:06.870297 3486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:06.876760 3486 catalog_manager.cc:1347] Generated new cluster ID: 1be93e41f9dc447a9d2c45c69089a61f
I20260620 20:15:06.876818 3486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:06.889816 3486 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:06.891675 3486 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:06.897449 3486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503: Generated new TSK 0
I20260620 20:15:06.898213 3486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:15:07.579083 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.20.148:35553/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:46011
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:07.695798 3517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:07.696033 3517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:07.696074 3517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:07.700016 3517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:07.700075 3517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:07.700131 3517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:15:07.705404 3517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.20.148:35553/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:46011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:07.706388 3517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:07.707392 3517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:07.717806 3523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:07.717813 3522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:07.718360 3517 server_base.cc:1034] running on GCE node
W20260620 20:15:07.718501 3525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:07.718964 3517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:07.719775 3517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:07.720960 3517 hybrid_clock.cc:648] HybridClock initialized: now 1781986507720944 us; error 31 us; skew 500 ppm
Jun 20 20:15:07 dist-test-slave-pflr krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986507, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:07.724251 3517 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:15:07.725193 3517 webserver.cc:460] Webserver started at http://127.24.20.129:44493/ using document root <none> and password file <none>
I20260620 20:15:07.725769 3517 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:07.725860 3517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:07.726092 3517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:15:07 dist-test-slave-pflr krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986507, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:07.747187 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "d12bf920c685491db2d2094dfe9861d9"
format_stamp: "Formatted at 2026-06-20 20:15:07 on dist-test-slave-pflr"
server_key: "020f318896d9470b9418589eb01c7920"
server_key_iv: "7ceaea47988319d61636664b1243a1fd"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:07.747972 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "d12bf920c685491db2d2094dfe9861d9"
format_stamp: "Formatted at 2026-06-20 20:15:07 on dist-test-slave-pflr"
server_key: "020f318896d9470b9418589eb01c7920"
server_key_iv: "7ceaea47988319d61636664b1243a1fd"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:07.753206 3517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:15:07.770211 3534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:07.771775 3517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:15:07.771889 3517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "d12bf920c685491db2d2094dfe9861d9"
format_stamp: "Formatted at 2026-06-20 20:15:07 on dist-test-slave-pflr"
server_key: "020f318896d9470b9418589eb01c7920"
server_key_iv: "7ceaea47988319d61636664b1243a1fd"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:07.772029 3517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:07.786842 3517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:07.790480 3517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:07.790720 3517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:07.791497 3517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:07.792532 3517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:07.792670 3517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:07.792791 3517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:07.792869 3517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:07.806660 3517 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:43231
I20260620 20:15:07.806679 3647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:43231 every 8 connection(s)
I20260620 20:15:07.807631 3517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:15:07.808506 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 3517
I20260620 20:15:07.808660 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:15:07.808992 24658 external_mini_cluster.cc:1428] Setting key 28251ba2bcf36d21be3272b49a36530a
Jun 20 20:15:07 dist-test-slave-pflr krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986507, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:15:07.827071 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:07.811385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:36341 (local address 127.24.20.190:46011)
0620 20:15:07.813026 (+ 1641us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:07.813035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:07.813053 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:15:07.813741 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:07.813747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:07.813807 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:07.813866 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:07.815857 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:07.816733 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:07.818711 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:07.819111 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:07.819193 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:07.822492 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:07.822497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:07.822500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:07.822527 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:07.824442 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:07.825174 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:07.825178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:07.825180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:07.825270 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:07.825717 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:07.825720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:07.825722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:07.825949 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:07.826039 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:07.826748 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 20:15:07.826879 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1558,"thread_start_us":98,"threads_started":1}
I20260620 20:15:07.828908 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:07.810340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:46011 (local address 127.24.20.129:36341)
0620 20:15:07.811819 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:07.811875 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:07.813491 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:07.814681 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:07.814690 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:07.815033 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:07.815691 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:07.815710 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:07.817003 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:07.817010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:07.818518 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:07.818531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:07.819455 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:07.819459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:07.819740 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:07.820513 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:07.820535 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:07.822269 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:07.824634 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:07.824641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:07.824659 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:07.825048 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:07.825387 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:07.825390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:07.825392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:07.825597 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:07.826112 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:07.826119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:07.826443 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 20:15:07.827259 (+ 816us) client_negotiation.cc:240] Negotiation successful
0620 20:15:07.827688 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"thread_start_us":133,"threads_started":1}
I20260620 20:15:07.830623 3648 heartbeater.cc:347] Connected to a master server at 127.24.20.190:46011
I20260620 20:15:07.830875 3648 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:07.831463 3648 heartbeater.cc:511] Master 127.24.20.190:46011 requested a full tablet report, sending...
I20260620 20:15:07.833329 3436 ts_manager.cc:194] Registered new tserver with Master: d12bf920c685491db2d2094dfe9861d9 (127.24.20.129:43231)
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:15:07.834959 3436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:36341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:15:07.875912 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.20.148:35553/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:46011
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:15:07 dist-test-slave-pflr krb5kdc[2960](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986507, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 20:15:08.011981 3656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:08.012262 3656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:08.012364 3656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:08.016405 3656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:08.016494 3656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:08.016664 3656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:15:08.023022 3656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.20.148:35553/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:46011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:08.024129 3656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:08.025388 3656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:08.035773 3665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:08.035774 3662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:08.036257 3656 server_base.cc:1034] running on GCE node
W20260620 20:15:08.036746 3663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:08.037154 3656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:08.037859 3656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:08.039032 3656 hybrid_clock.cc:648] HybridClock initialized: now 1781986508038999 us; error 46 us; skew 500 ppm
Jun 20 20:15:08 dist-test-slave-pflr krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986508, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:08.045979 3656 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:15:08.046967 3656 webserver.cc:460] Webserver started at http://127.24.20.130:38491/ using document root <none> and password file <none>
I20260620 20:15:08.047557 3656 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:08.047653 3656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:08.047948 3656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:15:08 dist-test-slave-pflr krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986508, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:08.063944 3656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "52e60c48113348dab9a4050267529151"
format_stamp: "Formatted at 2026-06-20 20:15:08 on dist-test-slave-pflr"
server_key: "4d5f8cbebd1ea32a7fd5db56b77d2817"
server_key_iv: "6509b1513e0a992d9e3b1a02e9f1cb41"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:08.064761 3656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "52e60c48113348dab9a4050267529151"
format_stamp: "Formatted at 2026-06-20 20:15:08 on dist-test-slave-pflr"
server_key: "4d5f8cbebd1ea32a7fd5db56b77d2817"
server_key_iv: "6509b1513e0a992d9e3b1a02e9f1cb41"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:08.069428 3656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 20:15:08.092967 3672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:08.094612 3656 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 20:15:08.094744 3656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "52e60c48113348dab9a4050267529151"
format_stamp: "Formatted at 2026-06-20 20:15:08 on dist-test-slave-pflr"
server_key: "4d5f8cbebd1ea32a7fd5db56b77d2817"
server_key_iv: "6509b1513e0a992d9e3b1a02e9f1cb41"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:08.094866 3656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:08.118053 3656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:08.121759 3656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:08.121999 3656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:08.122758 3656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:08.123657 3656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:08.123730 3656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:08.123828 3656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:08.123875 3656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:08.135386 3656 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:43559
I20260620 20:15:08.135527 3785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:43559 every 8 connection(s)
I20260620 20:15:08.136656 3656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:15:08.139448 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 3656
I20260620 20:15:08.139593 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:15:08.139909 24658 external_mini_cluster.cc:1428] Setting key 6775a6949734890055fff17c9d57023d
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:15:08.172578 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.20.148:35553/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:46011
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:38021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:15:08 dist-test-slave-pflr krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986508, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:15:08.179150 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.148725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:37617 (local address 127.24.20.190:46011)
0620 20:15:08.148850 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:08.148855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:08.165244 (+ 16389us) server_negotiation.cc:407] Connection header received
0620 20:15:08.166388 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:08.166391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:08.166465 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:08.166571 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:08.167903 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.168638 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.170083 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.170325 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.170452 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.173516 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:08.173523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:08.173529 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:08.173563 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:08.175482 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:08.176147 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:08.176153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:08.176158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:08.176248 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:08.176703 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:08.176709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:08.176713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:08.176977 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:08.177078 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:08.178717 (+ 1639us) server_negotiation.cc:299] Negotiation successful
0620 20:15:08.178871 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 20:15:08.180414 3789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.150081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:46011 (local address 127.24.20.130:37617)
0620 20:15:08.165044 (+ 14963us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:08.165119 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:08.166111 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:08.166784 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:08.166797 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:08.167164 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:08.167741 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.167755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.168772 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.168776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.169956 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.169974 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.170616 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.170619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.170807 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.171558 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:08.171583 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:08.173297 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:08.175628 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:08.175635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:08.175652 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:08.176029 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:08.176349 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:08.176355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:08.176360 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:08.176542 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:08.177136 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:08.177145 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:08.177478 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 20:15:08.177703 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 20:15:08.178089 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14608,"thread_start_us":230,"threads_started":1}
I20260620 20:15:08.181962 3786 heartbeater.cc:347] Connected to a master server at 127.24.20.190:46011
I20260620 20:15:08.182225 3786 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:08.182860 3786 heartbeater.cc:511] Master 127.24.20.190:46011 requested a full tablet report, sending...
I20260620 20:15:08.184134 3436 ts_manager.cc:194] Registered new tserver with Master: 52e60c48113348dab9a4050267529151 (127.24.20.130:43559)
I20260620 20:15:08.185058 3436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:37617
W20260620 20:15:08.320916 3791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:08.321295 3791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:08.321393 3791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:08.326102 3791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:08.326216 3791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:08.326330 3791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:15:08.332196 3791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.20.148:35553/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:46011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:08.333285 3791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:08.334337 3791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:08.345952 3797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:08.345952 3796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:08.346635 3799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:08.347792 3791 server_base.cc:1034] running on GCE node
I20260620 20:15:08.348189 3791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:08.348933 3791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:08.350147 3791 hybrid_clock.cc:648] HybridClock initialized: now 1781986508350071 us; error 87 us; skew 500 ppm
Jun 20 20:15:08 dist-test-slave-pflr krb5kdc[2960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986508, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:08.353158 3791 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:15:08.354182 3791 webserver.cc:460] Webserver started at http://127.24.20.131:46335/ using document root <none> and password file <none>
I20260620 20:15:08.354876 3791 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:08.354964 3791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:08.355202 3791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:15:08 dist-test-slave-pflr krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986508, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:08.381028 3791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "6d1908ecc30f48f4bdf15709aae80354"
format_stamp: "Formatted at 2026-06-20 20:15:08 on dist-test-slave-pflr"
server_key: "8590679b07948fe8189db600aa33e14c"
server_key_iv: "788efad043c5b100f55993d17534bede"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:08.382236 3791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "6d1908ecc30f48f4bdf15709aae80354"
format_stamp: "Formatted at 2026-06-20 20:15:08 on dist-test-slave-pflr"
server_key: "8590679b07948fe8189db600aa33e14c"
server_key_iv: "788efad043c5b100f55993d17534bede"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:08.386739 3791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:15:08.406931 3806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:08.408820 3791 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:15:08.408975 3791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "6d1908ecc30f48f4bdf15709aae80354"
format_stamp: "Formatted at 2026-06-20 20:15:08 on dist-test-slave-pflr"
server_key: "8590679b07948fe8189db600aa33e14c"
server_key_iv: "788efad043c5b100f55993d17534bede"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:08.409129 3791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 20:15:08 dist-test-slave-pflr krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986507, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:08.427872 3791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:08.431429 3791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:08.431660 3791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:08.432406 3791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:08.433755 3791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:08.433857 3791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:08.433970 3791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:08.434043 3791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:08.447005 3791 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:34831
I20260620 20:15:08.447974 3791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:15:08.449069 3919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:34831 every 8 connection(s)
I20260620 20:15:08.451781 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 3791
I20260620 20:15:08.451905 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:15:08.452193 24658 external_mini_cluster.cc:1428] Setting key afba4db12dbea5c232b79c2a8019cb66
Jun 20 20:15:08 dist-test-slave-pflr krb5kdc[2960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986508, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:15:08.468195 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.451698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:38089 (local address 127.24.20.190:46011)
0620 20:15:08.451840 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:08.451844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:08.451859 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:15:08.454976 (+ 3117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:08.454979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:08.455029 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:08.455076 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:08.456654 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.457435 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.459619 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.459944 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.460015 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.462893 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:08.462897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:08.462900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:08.462925 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:08.464712 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:08.466509 (+ 1797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:08.466514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:08.466518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:08.466606 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:08.466976 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:08.466981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:08.466982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:08.467197 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:08.467275 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:08.467914 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 20:15:08.468028 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:15:08.470101 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.450416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:46011 (local address 127.24.20.131:38089)
0620 20:15:08.450988 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:08.451046 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:08.454728 (+ 3682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:08.455486 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:08.455495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:08.455824 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:08.456480 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.456498 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.457578 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.457581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.459311 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.459324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.460219 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.460222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.460413 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.461083 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:08.461102 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:08.462722 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:08.466018 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:08.466024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:08.466041 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:08.466366 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:08.466706 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:08.466708 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:08.466710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:08.466874 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:08.467337 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:08.467343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:08.467653 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 20:15:08.468659 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0620 20:15:08.469013 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":145,"threads_started":1}
I20260620 20:15:08.471728 3920 heartbeater.cc:347] Connected to a master server at 127.24.20.190:46011
I20260620 20:15:08.472003 3920 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:08.472569 3920 heartbeater.cc:511] Master 127.24.20.190:46011 requested a full tablet report, sending...
I20260620 20:15:08.473843 3436 ts_manager.cc:194] Registered new tserver with Master: 6d1908ecc30f48f4bdf15709aae80354 (127.24.20.131:34831)
I20260620 20:15:08.474529 3436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:38089
I20260620 20:15:08.477596 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:15:08.491091 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.480861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.24.20.190:46011)
0620 20:15:08.480964 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:08.480969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:08.481858 (+ 889us) server_negotiation.cc:407] Connection header received
0620 20:15:08.482029 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:08.482034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:08.482079 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:08.482125 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:08.483171 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.483962 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.485190 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.485541 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.485659 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.487318 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:08.487322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:08.487325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:08.487351 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:08.489211 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:08.489701 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:08.489705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:08.489707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:08.489792 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:08.490127 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:08.490130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:08.490132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:08.490332 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:08.490430 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:08.490789 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 20:15:08.490917 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 20:15:08.494916 3436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:15:08.498939 3436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:15:08.531845 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.511539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:43559 (local address 127.0.0.1:41298)
0620 20:15:08.514861 (+ 3322us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:08.514873 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:08.514954 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:08.518035 (+ 3081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:08.518040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:08.518062 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:08.518180 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.518186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.526963 (+ 8777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.526966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.528804 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.528814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.531288 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.531292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.531414 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.531451 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:15:08.531551 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 20:15:08.531687 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3270,"thread_start_us":73,"threads_started":1}
I20260620 20:15:08.532330 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.512484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:43231 (local address 127.0.0.1:36140)
0620 20:15:08.516646 (+ 4162us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:08.516659 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:08.516741 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:08.518470 (+ 1729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:08.518472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:08.518485 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:08.518552 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.518567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.520983 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.520987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.522329 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.522340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.531923 (+ 9583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.531927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.532018 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.532053 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:15:08.532134 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 20:15:08.532204 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4044,"thread_start_us":114,"threads_started":1}
I20260620 20:15:08.533834 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.511891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.24.20.130:43559)
0620 20:15:08.514524 (+ 2633us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:08.514535 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:08.515044 (+ 509us) server_negotiation.cc:407] Connection header received
0620 20:15:08.515219 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:08.515225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:08.515432 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:08.515582 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:08.518852 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.526823 (+ 7971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.530215 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.530997 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.531099 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.531228 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:08.533600 (+ 2372us) server_negotiation.cc:299] Negotiation successful
0620 20:15:08.533699 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2534,"thread_start_us":75,"threads_started":1}
I20260620 20:15:08.534226 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.512893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36140 (local address 127.24.20.129:43231)
0620 20:15:08.514756 (+ 1863us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:08.514762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:08.516839 (+ 2077us) server_negotiation.cc:407] Connection header received
0620 20:15:08.516917 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:08.516923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:08.517109 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:08.517232 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:08.518697 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.520854 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.529025 (+ 8171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.529874 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.529978 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.530131 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:08.532759 (+ 2628us) server_negotiation.cc:299] Negotiation successful
0620 20:15:08.534097 (+ 1338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1769,"thread_start_us":80,"threads_started":1}
I20260620 20:15:08.534721 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.510273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.24.20.131:34831)
0620 20:15:08.517491 (+ 7218us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:08.517501 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:08.517521 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:15:08.517598 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:08.517603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:08.517767 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:08.517891 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:08.523515 (+ 5624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.524705 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.532501 (+ 7796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.533304 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.533410 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.533526 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:08.534490 (+ 964us) server_negotiation.cc:299] Negotiation successful
0620 20:15:08.534593 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7121,"thread_start_us":118,"threads_started":1}
I20260620 20:15:08.535681 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.509386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:34831 (local address 127.0.0.1:34230)
0620 20:15:08.514098 (+ 4712us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:08.514145 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:08.514261 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:08.518278 (+ 4017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:08.518281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:08.518294 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:08.518356 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.518360 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.524843 (+ 6483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.524846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.525928 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.525938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.533600 (+ 7662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.533604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.533692 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.533713 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:15:08.533784 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 20:15:08.533848 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4541,"thread_start_us":83,"threads_started":1}
I20260620 20:15:08.537230 3713 tablet_service.cc:1467] Processing CreateTablet for tablet de4eb52fc1764491bcc6a6551ec9a07b (DEFAULT_TABLE table=test-table [id=3e7eff8ae2ae4c2a9c2629dbed184589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:08.537582 3578 tablet_service.cc:1467] Processing CreateTablet for tablet de4eb52fc1764491bcc6a6551ec9a07b (DEFAULT_TABLE table=test-table [id=3e7eff8ae2ae4c2a9c2629dbed184589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:08.538357 3713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4eb52fc1764491bcc6a6551ec9a07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:08.538610 3578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4eb52fc1764491bcc6a6551ec9a07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:08.538291 3845 tablet_service.cc:1467] Processing CreateTablet for tablet de4eb52fc1764491bcc6a6551ec9a07b (DEFAULT_TABLE table=test-table [id=3e7eff8ae2ae4c2a9c2629dbed184589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:08.539178 3845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4eb52fc1764491bcc6a6551ec9a07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:08.548216 3941 tablet_bootstrap.cc:492] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354: Bootstrap starting.
I20260620 20:15:08.551105 3941 tablet_bootstrap.cc:654] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:08.551616 3939 tablet_bootstrap.cc:492] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9: Bootstrap starting.
I20260620 20:15:08.551906 3941 log.cc:824] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:08.552074 3940 tablet_bootstrap.cc:492] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151: Bootstrap starting.
I20260620 20:15:08.554292 3939 tablet_bootstrap.cc:654] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:08.555078 3939 log.cc:824] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:08.555528 3940 tablet_bootstrap.cc:654] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:08.556303 3940 log.cc:824] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:08.557147 3941 tablet_bootstrap.cc:492] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354: No bootstrap required, opened a new log
I20260620 20:15:08.557331 3941 ts_tablet_manager.cc:1397] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260620 20:15:08.558290 3939 tablet_bootstrap.cc:492] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9: No bootstrap required, opened a new log
I20260620 20:15:08.558446 3939 ts_tablet_manager.cc:1397] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 20:15:08.562712 3939 raft_consensus.cc:348] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1908ecc30f48f4bdf15709aae80354" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 34831 } } peers { permanent_uuid: "52e60c48113348dab9a4050267529151" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 43559 } } peers { permanent_uuid: "d12bf920c685491db2d2094dfe9861d9" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 43231 } }
I20260620 20:15:08.562714 3941 raft_consensus.cc:348] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1908ecc30f48f4bdf15709aae80354" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 34831 } } peers { permanent_uuid: "52e60c48113348dab9a4050267529151" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 43559 } } peers { permanent_uuid: "d12bf920c685491db2d2094dfe9861d9" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 43231 } }
I20260620 20:15:08.563098 3939 raft_consensus.cc:374] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:08.563186 3939 raft_consensus.cc:729] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12bf920c685491db2d2094dfe9861d9, State: Initialized, Role: FOLLOWER
I20260620 20:15:08.563453 3941 raft_consensus.cc:374] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:08.563522 3941 raft_consensus.cc:729] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1908ecc30f48f4bdf15709aae80354, State: Initialized, Role: FOLLOWER
I20260620 20:15:08.563990 3941 consensus_queue.cc:260] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1908ecc30f48f4bdf15709aae80354" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 34831 } } peers { permanent_uuid: "52e60c48113348dab9a4050267529151" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 43559 } } peers { permanent_uuid: "d12bf920c685491db2d2094dfe9861d9" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 43231 } }
I20260620 20:15:08.564781 3939 consensus_queue.cc:260] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1908ecc30f48f4bdf15709aae80354" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 34831 } } peers { permanent_uuid: "52e60c48113348dab9a4050267529151" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 43559 } } peers { permanent_uuid: "d12bf920c685491db2d2094dfe9861d9" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 43231 } }
I20260620 20:15:08.565094 3941 ts_tablet_manager.cc:1428] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 20:15:08.565452 3920 heartbeater.cc:503] Master 127.24.20.190:46011 was elected leader, sending a full tablet report...
I20260620 20:15:08.568118 3940 tablet_bootstrap.cc:492] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151: No bootstrap required, opened a new log
I20260620 20:15:08.568347 3940 ts_tablet_manager.cc:1397] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.002s
I20260620 20:15:08.568361 3648 heartbeater.cc:503] Master 127.24.20.190:46011 was elected leader, sending a full tablet report...
I20260620 20:15:08.572074 3940 raft_consensus.cc:348] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1908ecc30f48f4bdf15709aae80354" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 34831 } } peers { permanent_uuid: "52e60c48113348dab9a4050267529151" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 43559 } } peers { permanent_uuid: "d12bf920c685491db2d2094dfe9861d9" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 43231 } }
I20260620 20:15:08.572328 3940 raft_consensus.cc:374] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:08.572381 3940 raft_consensus.cc:729] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e60c48113348dab9a4050267529151, State: Initialized, Role: FOLLOWER
I20260620 20:15:08.572840 3940 consensus_queue.cc:260] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1908ecc30f48f4bdf15709aae80354" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 34831 } } peers { permanent_uuid: "52e60c48113348dab9a4050267529151" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 43559 } } peers { permanent_uuid: "d12bf920c685491db2d2094dfe9861d9" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 43231 } }
I20260620 20:15:08.573788 3940 ts_tablet_manager.cc:1428] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 20:15:08.574126 3786 heartbeater.cc:503] Master 127.24.20.190:46011 was elected leader, sending a full tablet report...
I20260620 20:15:08.574972 3939 ts_tablet_manager.cc:1428] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260620 20:15:08.640583 3947 raft_consensus.cc:482] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:15:08.640831 3947 raft_consensus.cc:504] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1908ecc30f48f4bdf15709aae80354" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 34831 } } peers { permanent_uuid: "52e60c48113348dab9a4050267529151" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 43559 } } peers { permanent_uuid: "d12bf920c685491db2d2094dfe9861d9" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 43231 } }
I20260620 20:15:08.643411 3947 leader_election.cc:288] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d1908ecc30f48f4bdf15709aae80354 (127.24.20.131:34831), d12bf920c685491db2d2094dfe9861d9 (127.24.20.129:43231)
W20260620 20:15:08.647298 3787 tablet.cc:2369] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:15:08.647578 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.642821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:52615 (local address 127.24.20.131:34831)
0620 20:15:08.642980 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:08.642987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:08.643005 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:15:08.643059 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:08.643062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:08.643109 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:08.643193 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:08.643662 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.644435 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.646159 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.646767 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.646875 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.646930 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:08.647350 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 20:15:08.647422 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:15:08.647830 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.643920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:48407 (local address 127.24.20.129:43231)
0620 20:15:08.644031 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:08.644035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:08.644047 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:15:08.644105 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:08.644108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:08.644147 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:08.644210 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:08.644530 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.645258 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.646634 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.647175 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.647256 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.647320 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:08.647648 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 20:15:08.647720 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:15:08.648190 3789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.642270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:34831 (local address 127.24.20.130:52615)
0620 20:15:08.642551 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:08.642574 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:08.642659 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:08.643318 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:08.643321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:08.643340 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:08.643453 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.643459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.644633 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.644637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.645966 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.645978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.646988 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.646991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.647087 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.647109 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:15:08.647182 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 20:15:08.647251 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260620 20:15:08.648209 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.643160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:43231 (local address 127.24.20.130:48407)
0620 20:15:08.643656 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:08.643669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:08.643749 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:08.644302 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:08.644305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:08.644319 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:08.644422 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.644427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.645392 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.645395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.646489 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:08.646499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.647432 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.647436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:08.647523 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.647541 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 20:15:08.647989 (+ 448us) client_negotiation.cc:240] Negotiation successful
0620 20:15:08.648056 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":78,"threads_started":1}
I20260620 20:15:08.648826 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4eb52fc1764491bcc6a6551ec9a07b" candidate_uuid: "52e60c48113348dab9a4050267529151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12bf920c685491db2d2094dfe9861d9" is_pre_election: true
I20260620 20:15:08.648825 3873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4eb52fc1764491bcc6a6551ec9a07b" candidate_uuid: "52e60c48113348dab9a4050267529151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1908ecc30f48f4bdf15709aae80354" is_pre_election: true
I20260620 20:15:08.649115 3586 raft_consensus.cc:2454] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52e60c48113348dab9a4050267529151 in term 0.
I20260620 20:15:08.649115 3873 raft_consensus.cc:2454] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52e60c48113348dab9a4050267529151 in term 0.
I20260620 20:15:08.649622 3676 leader_election.cc:302] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52e60c48113348dab9a4050267529151, d12bf920c685491db2d2094dfe9861d9; no voters:
I20260620 20:15:08.649904 3947 raft_consensus.cc:2781] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:15:08.650003 3947 raft_consensus.cc:482] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:15:08.650054 3947 raft_consensus.cc:3037] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:08.651371 3947 raft_consensus.cc:504] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1908ecc30f48f4bdf15709aae80354" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 34831 } } peers { permanent_uuid: "52e60c48113348dab9a4050267529151" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 43559 } } peers { permanent_uuid: "d12bf920c685491db2d2094dfe9861d9" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 43231 } }
I20260620 20:15:08.652037 3873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4eb52fc1764491bcc6a6551ec9a07b" candidate_uuid: "52e60c48113348dab9a4050267529151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d1908ecc30f48f4bdf15709aae80354"
I20260620 20:15:08.652045 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4eb52fc1764491bcc6a6551ec9a07b" candidate_uuid: "52e60c48113348dab9a4050267529151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12bf920c685491db2d2094dfe9861d9"
I20260620 20:15:08.652153 3873 raft_consensus.cc:3037] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:08.652158 3586 raft_consensus.cc:3037] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:08.653349 3873 raft_consensus.cc:2454] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52e60c48113348dab9a4050267529151 in term 1.
I20260620 20:15:08.653674 3673 leader_election.cc:302] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52e60c48113348dab9a4050267529151, 6d1908ecc30f48f4bdf15709aae80354; no voters:
I20260620 20:15:08.653767 3586 raft_consensus.cc:2454] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52e60c48113348dab9a4050267529151 in term 1.
I20260620 20:15:08.654075 3947 leader_election.cc:288] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [CANDIDATE]: Term 1 election: Requested vote from peers 6d1908ecc30f48f4bdf15709aae80354 (127.24.20.131:34831), d12bf920c685491db2d2094dfe9861d9 (127.24.20.129:43231)
I20260620 20:15:08.654208 3947 raft_consensus.cc:2781] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:08.654279 3947 raft_consensus.cc:686] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [term 1 LEADER]: Becoming Leader. State: Replica: 52e60c48113348dab9a4050267529151, State: Running, Role: LEADER
I20260620 20:15:08.654603 3947 consensus_queue.cc:237] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1908ecc30f48f4bdf15709aae80354" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 34831 } } peers { permanent_uuid: "52e60c48113348dab9a4050267529151" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 43559 } } peers { permanent_uuid: "d12bf920c685491db2d2094dfe9861d9" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 43231 } }
I20260620 20:15:08.657734 3436 catalog_manager.cc:5495] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52e60c48113348dab9a4050267529151 (127.24.20.130). New cstate: current_term: 1 leader_uuid: "52e60c48113348dab9a4050267529151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d1908ecc30f48f4bdf15709aae80354" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 34831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52e60c48113348dab9a4050267529151" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 43559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d12bf920c685491db2d2094dfe9861d9" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20260620 20:15:08.676532 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.671652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.24.20.130:43559)
0620 20:15:08.671757 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:08.671761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:08.671802 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:15:08.671965 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:08.671968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:08.672036 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:08.672146 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:08.672508 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.673295 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.674540 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:08.674865 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:08.674974 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:08.675302 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:08.675850 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:08.675980 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:08.676317 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 20:15:08.676374 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 20:15:08.683422 3586 raft_consensus.cc:1261] T de4eb52fc1764491bcc6a6551ec9a07b P d12bf920c685491db2d2094dfe9861d9 [term 1 FOLLOWER]: Refusing update from remote peer 52e60c48113348dab9a4050267529151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:08.683638 3873 raft_consensus.cc:1261] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354 [term 1 FOLLOWER]: Refusing update from remote peer 52e60c48113348dab9a4050267529151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:08.684273 3947 consensus_queue.cc:1035] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d1908ecc30f48f4bdf15709aae80354" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 34831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:08.684847 3947 consensus_queue.cc:1035] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12bf920c685491db2d2094dfe9861d9" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:08.696802 3952 mvcc.cc:205] Tried to move back new op lower bound from 7299016739560402944 to 7299016739451928576. Current Snapshot: MvccSnapshot[applied={T|T < 7299016739560402944 or (T in {7299016739560402944})}]
I20260620 20:15:08.699237 3955 mvcc.cc:205] Tried to move back new op lower bound from 7299016739560402944 to 7299016739451928576. Current Snapshot: MvccSnapshot[applied={T|T < 7299016739560402944}]
W20260620 20:15:08.703646 3921 tablet.cc:2369] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:15:08.705333 3954 mvcc.cc:205] Tried to move back new op lower bound from 7299016739560402944 to 7299016739451928576. Current Snapshot: MvccSnapshot[applied={T|T < 7299016739560402944}]
I20260620 20:15:08.706573 3436 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:08.706766 3436 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:08.709543 3436 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e14476a1a3534b08b5a85d44bd8af503: Sending DeleteTablet for 3 replicas of tablet de4eb52fc1764491bcc6a6551ec9a07b
I20260620 20:15:08.710629 3845 tablet_service.cc:1514] Processing DeleteTablet for tablet de4eb52fc1764491bcc6a6551ec9a07b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:08 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:34230
I20260620 20:15:08.711066 3578 tablet_service.cc:1514] Processing DeleteTablet for tablet de4eb52fc1764491bcc6a6551ec9a07b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:08 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:36140
I20260620 20:15:08.711911 3713 tablet_service.cc:1514] Processing DeleteTablet for tablet de4eb52fc1764491bcc6a6551ec9a07b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:08 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:41298
I20260620 20:15:08.713021 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 3517
I20260620 20:15:08.716746 3963 tablet_replica.cc:330] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354: stopping tablet replica
I20260620 20:15:08.717018 3963 raft_consensus.cc:2229] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:08.717305 3963 raft_consensus.cc:2258] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:08.719601 3965 tablet_replica.cc:330] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151: stopping tablet replica
I20260620 20:15:08.719892 3965 raft_consensus.cc:2229] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:15:08.720259 3965 raft_consensus.cc:2258] T de4eb52fc1764491bcc6a6551ec9a07b P 52e60c48113348dab9a4050267529151 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 20:15:08.723292 3423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:43231 (error 108)
I20260620 20:15:08.724092 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 3656
I20260620 20:15:08.726696 3963 ts_tablet_manager.cc:1905] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:08.730275 3963 ts_tablet_manager.cc:1918] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:15:08.730435 3963 log.cc:1196] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/de4eb52fc1764491bcc6a6551ec9a07b
I20260620 20:15:08.730837 3963 ts_tablet_manager.cc:1939] T de4eb52fc1764491bcc6a6551ec9a07b P 6d1908ecc30f48f4bdf15709aae80354: Deleting consensus metadata
I20260620 20:15:08.732720 3420 catalog_manager.cc:4841] TS 6d1908ecc30f48f4bdf15709aae80354 (127.24.20.131:34831): tablet de4eb52fc1764491bcc6a6551ec9a07b (table test-table [id=3e7eff8ae2ae4c2a9c2629dbed184589]) successfully deleted
I20260620 20:15:08.734526 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.731315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:43231 (local address 127.0.0.1:36152)
0620 20:15:08.734361 (+ 3046us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:08.734442 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:43231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2952}
W20260620 20:15:08.734746 3423 catalog_manager.cc:4584] TS d12bf920c685491db2d2094dfe9861d9 (127.24.20.129:43231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de4eb52fc1764491bcc6a6551ec9a07b: Network error: Client connection negotiation failed: client connection to 127.24.20.129:43231: connect: Connection refused (error 111)
I20260620 20:15:08.735015 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 3791
I20260620 20:15:08.737104 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:08.736882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:43559 (local address 127.0.0.1:41318)
0620 20:15:08.736991 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:08.737052 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.130:43559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 20:15:08.741205 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 3404
I20260620 20:15:08.747485 24658 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 20:15:09.082029 24658 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 20:15:09.082190 24658 mini_ranger.cc:66] Stopping Ranger...
I20260620 20:15:09.426608 24658 mini_ranger.cc:68] Stopped Ranger
2026-06-20 20:15:09.426 UTC [2990] LOG: received smart shutdown request
2026-06-20 20:15:09.430 UTC [2990] LOG: background worker "logical replication launcher" (PID 2999) exited with exit code 1
2026-06-20 20:15:09.430 UTC [2994] LOG: shutting down
2026-06-20 20:15:09.596 UTC [2990] LOG: database system is shut down
2026-06-20T20:15:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43984 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:09 dist-test-slave-pflr krb5kdc[3975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:09 dist-test-slave-pflr krb5kdc[3975](info): set up 2 sockets
Jun 20 20:15:09 dist-test-slave-pflr krb5kdc[3975](info): commencing operation
krb5kdc: starting...
W20260620 20:15:10.753482 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:15:10 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.20.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.20.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/rangeradmin_127.24.20.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/rangeradmin_127.24.20.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.20.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.20.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/rangerlookup_127.24.20.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/rangerlookup_127.24.20.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.20.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.20.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/HTTP_127.24.20.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/HTTP_127.24.20.148@KRBTEST.COM.keytab.
I20260620 20:15:10.923746 24658 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVWkq7u/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/postgres -l logfile start
2026-06-20 20:15:13.344 UTC [4005] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 20:15:13.344 UTC [4005] LOG: listening on IPv4 address "127.24.20.148", port 58163
2026-06-20 20:15:13.347 UTC [4005] LOG: listening on Unix socket "/tmp/.s.PGSQL.58163"
2026-06-20 20:15:13.368 UTC [4008] LOG: database system was shut down at 2026-06-20 20:15:11 UTC
2026-06-20 20:15:13.372 UTC [4005] LOG: database system is ready to accept connections
I20260620 20:15:14.328833 24658 mini_postgres.cc:96] Postgres bound to 58163
2026-06-20 20:15:14.335 UTC [4016] FATAL: database "slave" does not exist
127.24.20.148:58163 - accepting connections
I20260620 20:15:14.336174 24658 mini_ranger.cc:161] Starting Ranger...
I20260620 20:15:14.350684 24658 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 20:15:14.793269 24658 mini_ranger.cc:87] Created ranger Postgres database
I20260620 20:15:14.793354 24658 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-admin
2026-06-20 20:15:15,076 [I] DB FLAVOR :POSTGRES
2026-06-20 20:15:15,076 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:15:15,076 [I] Checking connection..
2026-06-20 20:15:15,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 20:15:15,306 [I] Checking connection passed.
2026-06-20 20:15:15,306 [I] --------- Verifying version history table ---------
2026-06-20 20:15:15,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:15:15,559 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:15:15,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:15:15,798 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:15:15,799 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 20:15:15,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 20:15:16.026 UTC [4080] WARNING: there is no transaction in progress
2026-06-20 20:15:16,032 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 20:15:16,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:15:16,259 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 20:15:16,260 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:15:16,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 20:15:16,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 20:15:16,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pflr.c.gcp-upstream.internal','N') ;"
2026-06-20 20:15:16,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:15:17,226 [I] Table x_portal_user does not exist in database ranger
2026-06-20 20:15:17,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:15:17,463 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 20:15:17,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:15:17,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:15:17,978 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 20:15:17,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 20:15:18.953 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:18.969 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:18.986 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19.003 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19.019 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19.035 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19.089 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19.096 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19.110 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19.119 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19.129 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19.141 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19.150 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19.156 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19.163 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19.170 UTC [4188] WARNING: there is no transaction in progress
2026-06-20 20:15:19,802 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 20:15:19,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 20:15:20,053 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 20:15:20,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:15:20,309 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 20:15:20,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:15:20,563 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 20:15:20,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:15:20,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:15:21,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pflr.c.gcp-upstream.internal';"
2026-06-20 20:15:21,271 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 20:15:21,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 20:15:21,528 [I] DB_PATCHES have already been applied
I20260620 20:15:21.534276 24658 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVWkq7u/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 20:15:21.534335 24658 mini_ranger.cc:193] Using host: 127.24.20.148
I20260620 20:15:21.537657 24658 mini_ranger.cc:227] Ranger admin URL: http://127.24.20.148:60171
Jun 20, 2026 8:15:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:15:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:15:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:15:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:15:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:15:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:15:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 20:15:22.648226 24658 mini_ranger.cc:160] Time spent starting Ranger: real 8.312s user 0.002s sys 0.007s
Jun 20, 2026 8:15:42 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:15:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 8:15:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:15:42 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:15:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:15:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:15:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:15:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:15:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:15:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:15:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:15:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:15:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:15:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:15:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:15:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:15:42 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:15:43 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 20:15:44.087450 24658 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.20.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.20.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/rangerkms_127.24.20.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.20.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/rangerkms_127.24.20.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.20.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.20.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.20.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/HTTP_127.24.20.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.20.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/HTTP_127.24.20.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 20:15:44 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986544, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 20:15:44.169515 24658 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 20:15:44.178733 24658 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 20:15:45.004647 24658 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 20:15:45.004724 24658 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-kms
I20260620 20:15:45.004737 24658 mini_ranger_kms.cc:227] Using postgres at 127.24.20.148:58163
2026-06-20 20:15:45,182 [I] DB FLAVOR :POSTGRES
2026-06-20 20:15:45,183 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:15:45,183 [I] Checking connection
2026-06-20 20:15:45,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 20:15:45,406 [I] connection success
2026-06-20 20:15:45,406 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 20:15:45,406 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 20:15:45,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 20:15:45,636 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 20:15:45,636 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:15:45,636 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 20:15:45,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.20.148:58163/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 20:15:45,880 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 20:15:46.328896 24658 mini_ranger_kms.cc:310] Created kms service
I20260620 20:15:46.506937 24658 mini_ranger_kms.cc:326] Created kudu user
I20260620 20:15:46.565361 24658 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 20:15:46 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986544, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:46.989228 24658 mini_ranger_kms.cc:384] Added ranger policy
I20260620 20:15:46.989383 24658 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVWkq7u/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVWkq7u/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 20:15:46.989406 24658 mini_ranger_kms.cc:240] Using host: 127.24.20.148
I20260620 20:15:46.992913 24658 mini_ranger_kms.cc:276] Ranger KMS PID: 4368
I20260620 20:15:46.992998 24658 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.20.148:43913
Jun 20, 2026 8:15:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:15:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:15:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:15:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:15:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:15:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 20:15:49 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986549, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.20.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:49.265344 24658 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.096s user 0.014s sys 0.148s
I20260620 20:15:49.265473 24658 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 20:15:49.265537 24658 mini_ranger_kms.cc:401] 127.24.20.148:43913/kms/v1/keys
Jun 20 20:15:49 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986549, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.20.148@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
Jun 20 20:15:49 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986549, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.20.148@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
Jun 20 20:15:52 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:15:52.129017 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:34815
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:34815
--ranger_config_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.20.148:43913/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:52.238612 4419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:52.238848 4419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:52.238884 4419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:52.242875 4419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:52.242928 4419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:52.242947 4419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:52.242964 4419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:52.243018 4419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:52.248291 4419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.20.148:43913/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:34815
--ranger_config_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:34815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:52.249287 4419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:52.250265 4419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:52.258635 4427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:52.258631 4424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:52.258628 4425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:52.258780 4419 server_base.cc:1034] running on GCE node
I20260620 20:15:52.259411 4419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:52.260430 4419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:52.261590 4419 hybrid_clock.cc:648] HybridClock initialized: now 1781986552261597 us; error 55 us; skew 500 ppm
Jun 20 20:15:52 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986552, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:52.267336 4419 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:15:52.268185 4419 webserver.cc:460] Webserver started at http://127.24.20.190:41027/ using document root <none> and password file <none>
I20260620 20:15:52.268771 4419 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:52.268831 4419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:52.269062 4419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:15:52 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986552, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:52.316108 4419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "a41d7e966c8c423cbe63e26dca88b12e"
format_stamp: "Formatted at 2026-06-20 20:15:52 on dist-test-slave-pflr"
server_key: "ba4925905a671fc718ff8f3e4cb0999f"
server_key_iv: "f4e9e4823bd42c7be5e3c356e57d5ede"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:52.316787 4419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "a41d7e966c8c423cbe63e26dca88b12e"
format_stamp: "Formatted at 2026-06-20 20:15:52 on dist-test-slave-pflr"
server_key: "ba4925905a671fc718ff8f3e4cb0999f"
server_key_iv: "f4e9e4823bd42c7be5e3c356e57d5ede"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:52.320921 4419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 20:15:52.334193 4434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:52.335541 4419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:15:52.335676 4419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "a41d7e966c8c423cbe63e26dca88b12e"
format_stamp: "Formatted at 2026-06-20 20:15:52 on dist-test-slave-pflr"
server_key: "ba4925905a671fc718ff8f3e4cb0999f"
server_key_iv: "f4e9e4823bd42c7be5e3c356e57d5ede"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:52.335795 4419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:52.345203 4419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:52.348155 4419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:52.348351 4419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:52.355532 4419 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:34815
I20260620 20:15:52.355559 4486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:34815 every 8 connection(s)
I20260620 20:15:52.356451 4419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:52.359812 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:52.365442 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 4419
I20260620 20:15:52.365607 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:15:52.365907 24658 external_mini_cluster.cc:1428] Setting key 90630fba704d35ed32d5a514669ab3b5
I20260620 20:15:52.366626 4487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e: Bootstrap starting.
I20260620 20:15:52.369398 4487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:52.370345 4487 log.cc:824] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:52.372490 4487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e: No bootstrap required, opened a new log
Jun 20 20:15:52 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:15:52.376911 4487 raft_consensus.cc:348] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d7e966c8c423cbe63e26dca88b12e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34815 } }
I20260620 20:15:52.377259 4487 raft_consensus.cc:374] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:52.377327 4487 raft_consensus.cc:729] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41d7e966c8c423cbe63e26dca88b12e, State: Initialized, Role: FOLLOWER
I20260620 20:15:52.377780 4487 consensus_queue.cc:260] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d7e966c8c423cbe63e26dca88b12e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34815 } }
I20260620 20:15:52.377931 4487 raft_consensus.cc:388] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:52.378028 4487 raft_consensus.cc:482] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:52.378125 4487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:52.379134 4487 raft_consensus.cc:504] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d7e966c8c423cbe63e26dca88b12e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34815 } }
I20260620 20:15:52.379509 4487 leader_election.cc:302] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a41d7e966c8c423cbe63e26dca88b12e; no voters:
I20260620 20:15:52.379807 4490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:52.367364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.24.20.190:34815)
0620 20:15:52.367762 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:52.367801 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:52.367833 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:15:52.368436 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:52.368451 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:52.368826 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:52.369157 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:52.369928 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.371118 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.371971 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:52.372437 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:52.372597 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:52.375058 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:52.375063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:52.375075 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:52.375099 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:52.376778 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:52.377210 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:52.377216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:52.377222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:52.377347 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:52.377711 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:52.377713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:52.377714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:52.378193 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:52.378390 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:52.378671 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 20:15:52.378950 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":118,"threads_started":1}
I20260620 20:15:52.379818 4487 leader_election.cc:288] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:15:52.379909 4492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:52.380425 4492 raft_consensus.cc:686] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 1 LEADER]: Becoming Leader. State: Replica: a41d7e966c8c423cbe63e26dca88b12e, State: Running, Role: LEADER
I20260620 20:15:52.380775 4492 consensus_queue.cc:237] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d7e966c8c423cbe63e26dca88b12e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34815 } }
I20260620 20:15:52.381397 4487 sys_catalog.cc:564] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:52.381937 4492 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a41d7e966c8c423cbe63e26dca88b12e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d7e966c8c423cbe63e26dca88b12e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34815 } } }
I20260620 20:15:52.382046 4492 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:52.382246 4492 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a41d7e966c8c423cbe63e26dca88b12e. Latest consensus state: current_term: 1 leader_uuid: "a41d7e966c8c423cbe63e26dca88b12e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d7e966c8c423cbe63e26dca88b12e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34815 } } }
I20260620 20:15:52.382328 4492 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:52.384052 4487 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:15:52.384982 4496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:52.390728 4496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:52.397548 4496 catalog_manager.cc:1347] Generated new cluster ID: 258d0305b1404c709534bb9cc6737b16
I20260620 20:15:52.398886 4496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:52.417299 4496 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:52.418254 4496 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:52.426170 4496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e: Generated new TSK 0
I20260620 20:15:52.426828 4496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:15:53.115790 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.20.148:43913/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:34815
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:53.270985 4532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:53.271291 4532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:53.271395 4532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:53.275662 4532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:53.275758 4532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:53.275879 4532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:15:53.283272 4532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.20.148:43913/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:34815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:53.284828 4532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:53.286281 4532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:15:53.300720 4532 server_base.cc:1034] running on GCE node
W20260620 20:15:53.301232 4540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:53.301632 4539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:53.301975 4542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:53.302573 4532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:53.303406 4532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:53.304666 4532 hybrid_clock.cc:648] HybridClock initialized: now 1781986553304588 us; error 96 us; skew 500 ppm
Jun 20 20:15:53 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986553, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:53.309168 4532 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:15:53.310426 4532 webserver.cc:460] Webserver started at http://127.24.20.129:44229/ using document root <none> and password file <none>
I20260620 20:15:53.311151 4532 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:53.311216 4532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:53.311444 4532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:15:53 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986553, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:53.327281 4532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "996f446b7ebd4944a0ff16d2486c1bba"
format_stamp: "Formatted at 2026-06-20 20:15:53 on dist-test-slave-pflr"
server_key: "a0517e8778157b3293dc820a73322d76"
server_key_iv: "41411dc8fe51a59d35686aa4b5c8e511"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:53.327996 4532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "996f446b7ebd4944a0ff16d2486c1bba"
format_stamp: "Formatted at 2026-06-20 20:15:53 on dist-test-slave-pflr"
server_key: "a0517e8778157b3293dc820a73322d76"
server_key_iv: "41411dc8fe51a59d35686aa4b5c8e511"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:53.340045 4532 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.007s sys 0.000s
I20260620 20:15:53.368254 4549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:53.370033 4532 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.002s
I20260620 20:15:53.370167 4532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "996f446b7ebd4944a0ff16d2486c1bba"
format_stamp: "Formatted at 2026-06-20 20:15:53 on dist-test-slave-pflr"
server_key: "a0517e8778157b3293dc820a73322d76"
server_key_iv: "41411dc8fe51a59d35686aa4b5c8e511"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:53.370317 4532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:53.410099 4532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:53.413815 4532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:53.414002 4532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:53.414702 4532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:53.415696 4532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:53.415756 4532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:53.415808 4532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:53.415886 4532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:53.426797 4532 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:37217
I20260620 20:15:53.427860 4532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:15:53.429467 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 4532
I20260620 20:15:53.429587 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:15:53.429901 24658 external_mini_cluster.cc:1428] Setting key 8a7b54ad523f5118b9f6a8205918075c
I20260620 20:15:53.428705 4664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:37217 every 8 connection(s)
Jun 20 20:15:53 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986553, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
I20260620 20:15:53.447800 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.431343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:33271 (local address 127.24.20.190:34815)
0620 20:15:53.431715 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:53.431720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:53.433379 (+ 1659us) server_negotiation.cc:407] Connection header received
0620 20:15:53.434760 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:53.434763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:53.434821 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:53.434888 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:53.436499 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.437390 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.439236 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.439660 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.439768 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.443011 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:53.443015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:53.443018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:53.443045 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:53.445143 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:53.445895 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:53.445899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:53.445900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:53.445996 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:53.446464 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:53.446470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:53.446471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:53.446732 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:53.446835 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:53.447468 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 20:15:53.447608 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":170,"threads_started":1}
I20260620 20:15:53.449733 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.431770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:34815 (local address 127.24.20.129:33271)
0620 20:15:53.433231 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:53.433287 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:53.434512 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:53.435300 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:53.435309 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:53.435647 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:53.436310 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.436333 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.437567 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.437571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.439085 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.439098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.440061 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.440066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.440264 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.440977 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:53.440999 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:53.442810 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:53.445399 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:53.445406 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:53.445422 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:53.445769 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:53.446110 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:53.446113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:53.446116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:53.446311 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:53.446859 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:53.446866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:53.447170 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 20:15:53.448260 (+ 1090us) client_negotiation.cc:240] Negotiation successful
0620 20:15:53.448669 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"thread_start_us":151,"threads_started":1}
I20260620 20:15:53.451030 4665 heartbeater.cc:347] Connected to a master server at 127.24.20.190:34815
I20260620 20:15:53.451285 4665 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:53.451843 4665 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
I20260620 20:15:53.453588 4451 ts_manager.cc:194] Registered new tserver with Master: 996f446b7ebd4944a0ff16d2486c1bba (127.24.20.129:37217)
I20260620 20:15:53.455087 4451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:33271
Jun 20 20:15:53 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986553, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:15:53.473774 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.20.148:43913/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:34815
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:53.630331 4671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:53.630702 4671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:53.630771 4671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:53.636657 4671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:53.636777 4671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:53.636889 4671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:15:53.644934 4671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.20.148:43913/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:34815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:53.646389 4671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:53.647759 4671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:53.660231 4678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:53.660216 4677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:53.660450 4680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:53.661736 4671 server_base.cc:1034] running on GCE node
I20260620 20:15:53.662165 4671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:53.662890 4671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:53.664067 4671 hybrid_clock.cc:648] HybridClock initialized: now 1781986553664002 us; error 77 us; skew 500 ppm
Jun 20 20:15:53 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986553, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:53.667714 4671 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:15:53.668788 4671 webserver.cc:460] Webserver started at http://127.24.20.130:42091/ using document root <none> and password file <none>
I20260620 20:15:53.669644 4671 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:53.669741 4671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:53.670009 4671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:15:53 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986553, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:53.683830 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "4bdce36f709d460f851b2170b85e96bc"
format_stamp: "Formatted at 2026-06-20 20:15:53 on dist-test-slave-pflr"
server_key: "86516df7d68724e1098379cf2fe2c296"
server_key_iv: "dc34b6305ce4a78ea468ac88021f37ce"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:53.684648 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "4bdce36f709d460f851b2170b85e96bc"
format_stamp: "Formatted at 2026-06-20 20:15:53 on dist-test-slave-pflr"
server_key: "86516df7d68724e1098379cf2fe2c296"
server_key_iv: "dc34b6305ce4a78ea468ac88021f37ce"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:53.689746 4671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260620 20:15:53.705452 4687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:53.707289 4671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:15:53.707423 4671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "4bdce36f709d460f851b2170b85e96bc"
format_stamp: "Formatted at 2026-06-20 20:15:53 on dist-test-slave-pflr"
server_key: "86516df7d68724e1098379cf2fe2c296"
server_key_iv: "dc34b6305ce4a78ea468ac88021f37ce"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:53.707621 4671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:53.732167 4671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:53.735846 4671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:53.736106 4671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:53.736872 4671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:53.737953 4671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:53.738071 4671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:53.738181 4671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:53.738256 4671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:53.749249 4671 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:37133
I20260620 20:15:53.750365 4671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:15:53.752431 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 4671
I20260620 20:15:53.752553 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:15:53.752894 24658 external_mini_cluster.cc:1428] Setting key ac7b47ddfcad0ecb23a953e505c8e8bc
I20260620 20:15:53.760528 4800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:37133 every 8 connection(s)
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
Jun 20 20:15:53 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986553, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:15:53.789770 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.763066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:37627 (local address 127.24.20.190:34815)
0620 20:15:53.763215 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:53.763220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:53.769018 (+ 5798us) server_negotiation.cc:407] Connection header received
0620 20:15:53.770243 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:53.770247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:53.770307 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:53.770403 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:53.775989 (+ 5586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.776936 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.781005 (+ 4069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.781447 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.781544 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.785070 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:53.785077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:53.785083 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:53.785119 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:53.787026 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:53.787696 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:53.787702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:53.787704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:53.787807 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:53.788248 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:53.788253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:53.788257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:53.788524 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:53.788639 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:53.789456 (+ 817us) server_negotiation.cc:299] Negotiation successful
0620 20:15:53.789594 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:15:53.791383 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:53.763590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:34815 (local address 127.24.20.130:37627)
0620 20:15:53.768837 (+ 5247us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:53.768901 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:53.769999 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:53.772864 (+ 2865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:53.772874 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:53.773258 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:53.775125 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.775148 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.777238 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.777241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.780835 (+ 3594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:53.780848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:53.781788 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:53.781793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:53.782038 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:53.782885 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:53.782912 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:53.784842 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:53.787170 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:53.787178 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:53.787199 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:53.787570 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:53.787922 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:53.787927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:53.787929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:53.788124 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:53.788780 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:53.788786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:53.789137 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 20:15:53.789952 (+ 815us) client_negotiation.cc:240] Negotiation successful
0620 20:15:53.790310 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4943,"thread_start_us":141,"threads_started":1}
I20260620 20:15:53.792776 4801 heartbeater.cc:347] Connected to a master server at 127.24.20.190:34815
I20260620 20:15:53.793020 4801 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:53.793629 4801 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
I20260620 20:15:53.794765 4451 ts_manager.cc:194] Registered new tserver with Master: 4bdce36f709d460f851b2170b85e96bc (127.24.20.130:37133)
I20260620 20:15:53.795420 4451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:37627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:15:53.800814 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.20.148:43913/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:34815
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:15:53 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986553, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
W20260620 20:15:53.958308 4806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:53.958786 4806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:53.958911 4806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:53.965907 4806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:53.966086 4806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:53.966243 4806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:15:53.974951 4806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.20.148:43913/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:34815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:53.976617 4806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:53.978039 4806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:53.991662 4811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:53.992776 4806 server_base.cc:1034] running on GCE node
W20260620 20:15:53.992631 4814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:53.995577 4812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:53.996106 4806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:53.996872 4806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:53.998055 4806 hybrid_clock.cc:648] HybridClock initialized: now 1781986553997992 us; error 79 us; skew 500 ppm
Jun 20 20:15:53 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986553, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:54.001751 4806 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:15:54.002909 4806 webserver.cc:460] Webserver started at http://127.24.20.131:39507/ using document root <none> and password file <none>
I20260620 20:15:54.003633 4806 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:54.003767 4806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:54.004043 4806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:15:54 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986553, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:54.030002 4806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "cf66a651366349e783b8ccbf51ee8871"
format_stamp: "Formatted at 2026-06-20 20:15:54 on dist-test-slave-pflr"
server_key: "fe816737e0e0ed47e75ba4dbed17fcb9"
server_key_iv: "7495ea4ee14ed7c193bf69a3cd8245b7"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:54.030831 4806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "cf66a651366349e783b8ccbf51ee8871"
format_stamp: "Formatted at 2026-06-20 20:15:54 on dist-test-slave-pflr"
server_key: "fe816737e0e0ed47e75ba4dbed17fcb9"
server_key_iv: "7495ea4ee14ed7c193bf69a3cd8245b7"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:54.043622 4806 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.002s sys 0.004s
I20260620 20:15:54.063325 4821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:54.065032 4806 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 20:15:54.065155 4806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "cf66a651366349e783b8ccbf51ee8871"
format_stamp: "Formatted at 2026-06-20 20:15:54 on dist-test-slave-pflr"
server_key: "fe816737e0e0ed47e75ba4dbed17fcb9"
server_key_iv: "7495ea4ee14ed7c193bf69a3cd8245b7"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:54.065277 4806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:54.092789 4806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:54.095788 4806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:54.095985 4806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:54.096586 4806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:54.097505 4806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:54.097584 4806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:54.097703 4806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:54.097750 4806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:54.108357 4806 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:32849
I20260620 20:15:54.108385 4934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:32849 every 8 connection(s)
I20260620 20:15:54.109283 4806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:15:54.109952 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 4806
I20260620 20:15:54.110098 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:15:54.110363 24658 external_mini_cluster.cc:1428] Setting key d4ab4d1dcacac76dcd718ef1c73dd693
Jun 20 20:15:54 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986553, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:15:54.125190 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.112257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:59981 (local address 127.24.20.190:34815)
0620 20:15:54.112371 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:54.112375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:54.113445 (+ 1070us) server_negotiation.cc:407] Connection header received
0620 20:15:54.114708 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:54.114711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:54.114764 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:54.114866 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:54.115870 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.116393 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.118030 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.118307 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.118417 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.121039 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:54.121042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:54.121045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:54.121068 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:54.122565 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:54.123369 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:54.123372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:54.123374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:54.123485 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:54.123895 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:54.123898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:54.123901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:54.124113 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:54.124199 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:54.124896 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 20:15:54.125022 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:15:54.126619 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.112871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:34815 (local address 127.24.20.131:59981)
0620 20:15:54.113312 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:54.113360 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:54.114496 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:54.115033 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:54.115040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:54.115286 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:54.115736 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.115749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.116512 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.116516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.117908 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.117918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.118447 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.118452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.118623 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.119196 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:54.119235 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:54.120881 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:54.122696 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:54.122701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:54.122714 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:54.123256 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:54.123603 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:54.123606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:54.123608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:54.123779 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:54.124269 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:54.124275 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:54.124587 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 20:15:54.125328 (+ 741us) client_negotiation.cc:240] Negotiation successful
0620 20:15:54.125648 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":129,"threads_started":1}
I20260620 20:15:54.127905 4935 heartbeater.cc:347] Connected to a master server at 127.24.20.190:34815
I20260620 20:15:54.128134 4935 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:54.128526 4935 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
I20260620 20:15:54.129455 4451 ts_manager.cc:194] Registered new tserver with Master: cf66a651366349e783b8ccbf51ee8871 (127.24.20.131:32849)
I20260620 20:15:54.129993 4451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:59981
I20260620 20:15:54.140951 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:15:54.141695 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 4532
I20260620 20:15:54.147710 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 4671
I20260620 20:15:54.154608 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 4806
I20260620 20:15:54.159343 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 4419
I20260620 20:15:54.165225 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:34815
--webserver_interface=127.24.20.190
--webserver_port=41027
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:34815
--ranger_config_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.20.148:43913/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.20.148:43913/kms with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:54.273713 4938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:54.273968 4938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:54.274006 4938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:54.277937 4938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:54.277989 4938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:54.278008 4938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:54.278024 4938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:54.278039 4938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:54.283186 4938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.20.148:43913/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:34815
--ranger_config_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:34815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=41027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:54.284032 4938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:54.284912 4938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:54.292865 4944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:54.292868 4946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:54.292900 4943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:54.293247 4938 server_base.cc:1034] running on GCE node
I20260620 20:15:54.293751 4938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:54.294826 4938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:54.296020 4938 hybrid_clock.cc:648] HybridClock initialized: now 1781986554296009 us; error 30 us; skew 500 ppm
Jun 20 20:15:54 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986554, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:54.298734 4938 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:15:54.299580 4938 webserver.cc:460] Webserver started at http://127.24.20.190:41027/ using document root <none> and password file <none>
I20260620 20:15:54.300101 4938 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:54.300217 4938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:15:54 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986554, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:54.379912 4938 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:15:54.382151 4954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:54.383431 4938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:15:54.383592 4938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "a41d7e966c8c423cbe63e26dca88b12e"
format_stamp: "Formatted at 2026-06-20 20:15:52 on dist-test-slave-pflr"
server_key: "ba4925905a671fc718ff8f3e4cb0999f"
server_key_iv: "f4e9e4823bd42c7be5e3c356e57d5ede"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:54.384153 4938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:54.393923 4938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:54.396257 4938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:54.396802 4938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:54.403867 4938 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:34815
I20260620 20:15:54.403878 5007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:34815 every 8 connection(s)
I20260620 20:15:54.404915 4938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:54.409032 5008 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:15:54.410580 5008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e: Bootstrap starting.
I20260620 20:15:54.412024 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 4938
I20260620 20:15:54.412526 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.20.148:43913/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:37217
--local_ip_for_outbound_sockets=127.24.20.129
--tserver_master_addrs=127.24.20.190:34815
--webserver_port=44229
--webserver_interface=127.24.20.129
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.20.148:43913/kms with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:15:54.426669 5008 log.cc:824] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:54.433247 5008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:15:54.433679 5008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e: Bootstrap complete.
I20260620 20:15:54.437798 5008 raft_consensus.cc:348] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d7e966c8c423cbe63e26dca88b12e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34815 } }
I20260620 20:15:54.438205 5008 raft_consensus.cc:729] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41d7e966c8c423cbe63e26dca88b12e, State: Initialized, Role: FOLLOWER
I20260620 20:15:54.438717 5008 consensus_queue.cc:260] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d7e966c8c423cbe63e26dca88b12e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34815 } }
I20260620 20:15:54.438863 5008 raft_consensus.cc:388] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:15:54.438951 5008 raft_consensus.cc:482] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:15:54.439075 5008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:15:54.440987 5008 raft_consensus.cc:504] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d7e966c8c423cbe63e26dca88b12e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34815 } }
I20260620 20:15:54.441298 5008 leader_election.cc:302] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a41d7e966c8c423cbe63e26dca88b12e; no voters:
I20260620 20:15:54.441635 5008 leader_election.cc:288] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:15:54.441695 5012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:15:54.441887 5012 raft_consensus.cc:686] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [term 2 LEADER]: Becoming Leader. State: Replica: a41d7e966c8c423cbe63e26dca88b12e, State: Running, Role: LEADER
I20260620 20:15:54.442216 5012 consensus_queue.cc:237] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d7e966c8c423cbe63e26dca88b12e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34815 } }
I20260620 20:15:54.442826 5008 sys_catalog.cc:564] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:54.443509 5013 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a41d7e966c8c423cbe63e26dca88b12e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d7e966c8c423cbe63e26dca88b12e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34815 } } }
I20260620 20:15:54.443645 5013 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:54.444342 5019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:54.444425 5014 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a41d7e966c8c423cbe63e26dca88b12e. Latest consensus state: current_term: 2 leader_uuid: "a41d7e966c8c423cbe63e26dca88b12e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41d7e966c8c423cbe63e26dca88b12e" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34815 } } }
I20260620 20:15:54.444530 5014 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:54.447562 5019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:54.448062 5008 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:15:54.448210 5019 catalog_manager.cc:1259] Loaded cluster ID: 258d0305b1404c709534bb9cc6737b16
I20260620 20:15:54.448257 5019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:54.450528 5019 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:54.451155 5019 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e: Loaded TSK: 0
I20260620 20:15:54.452286 5019 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:15:54.534088 5010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:54.534443 5010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:54.534526 5010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:54.541122 5010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:54.541198 5010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:54.541287 5010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:15:54.549432 5010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.20.148:43913/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:37217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=44229
--tserver_master_addrs=127.24.20.190:34815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:54.550866 5010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:54.552280 5010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:54.564357 5047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:54.564388 5048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:54.564432 5050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:54.564976 5010 server_base.cc:1034] running on GCE node
I20260620 20:15:54.565404 5010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:54.566133 5010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:54.567354 5010 hybrid_clock.cc:648] HybridClock initialized: now 1781986554567303 us; error 40 us; skew 500 ppm
Jun 20 20:15:54 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986554, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:54.570847 5010 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:15:54.571868 5010 webserver.cc:460] Webserver started at http://127.24.20.129:44229/ using document root <none> and password file <none>
I20260620 20:15:54.572525 5010 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:54.572641 5010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:15:54 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986554, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:54.600142 5010 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:15:54.603379 5058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:54.605007 5010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:15:54.605125 5010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "996f446b7ebd4944a0ff16d2486c1bba"
format_stamp: "Formatted at 2026-06-20 20:15:53 on dist-test-slave-pflr"
server_key: "a0517e8778157b3293dc820a73322d76"
server_key_iv: "41411dc8fe51a59d35686aa4b5c8e511"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:54.605829 5010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:54.620524 5010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:54.624133 5010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:54.624325 5010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:54.624986 5010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:54.625938 5010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:54.625994 5010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:54.626041 5010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:54.626101 5010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:54.635797 5010 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:37217
I20260620 20:15:54.637056 5010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:15:54.641053 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 5010
I20260620 20:15:54.641566 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.20.148:43913/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:37133
--local_ip_for_outbound_sockets=127.24.20.130
--tserver_master_addrs=127.24.20.190:34815
--webserver_port=42091
--webserver_interface=127.24.20.130
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.20.148:43913/kms with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:15:54.660666 5172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:37217 every 8 connection(s)
Jun 20 20:15:54 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986554, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:15:54.692839 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.661606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:34815 (local address 127.24.20.129:39063)
0620 20:15:54.671150 (+ 9544us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:54.671211 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:54.672240 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:54.674102 (+ 1862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:54.674110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:54.674450 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:54.675029 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.675047 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.677334 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.677338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.678598 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.678609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.679784 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.679788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.679962 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.685148 (+ 5186us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:54.685169 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:54.686961 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:54.689480 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:54.689486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:54.689498 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:54.689813 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:54.690209 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:54.690212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:54.690214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:54.690366 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:54.691088 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:54.691093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:54.691410 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 20:15:54.691626 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 20:15:54.691915 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9304,"thread_start_us":137,"threads_started":1}
I20260620 20:15:54.694437 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.663095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:39063 (local address 127.24.20.190:34815)
0620 20:15:54.669984 (+ 6889us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:54.670028 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:54.672549 (+ 2521us) server_negotiation.cc:407] Connection header received
0620 20:15:54.672858 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:54.672872 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:54.673275 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:54.673610 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:54.675229 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.677145 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.678733 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.679440 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.679586 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.687199 (+ 7613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:54.687206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:54.687221 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:54.687254 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:54.689346 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:54.689946 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:54.689951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:54.689957 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:54.690100 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:54.690468 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:54.690471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:54.690473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:54.690899 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:54.691011 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:54.693330 (+ 2319us) server_negotiation.cc:299] Negotiation successful
0620 20:15:54.693601 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6273,"thread_start_us":100,"threads_started":1}
I20260620 20:15:54.696969 5173 heartbeater.cc:347] Connected to a master server at 127.24.20.190:34815
I20260620 20:15:54.697197 5173 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:54.697721 5173 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
W20260620 20:15:54.699851 5173 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:34815 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:15:54.699965 5173 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:54.700122 5173 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
I20260620 20:15:54.701016 5173 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:54.701174 5173 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
W20260620 20:15:54.701954 5173 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:15:54.798360 5175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:54.798723 5175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:54.798815 5175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:54.805078 5175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:54.805159 5175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:54.805246 5175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:15:54.813238 5175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.20.148:43913/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:37133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=42091
--tserver_master_addrs=127.24.20.190:34815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:54.814702 5175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:54.816116 5175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:54.828542 5182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:54.828806 5183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:54.828567 5185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:54.829134 5175 server_base.cc:1034] running on GCE node
I20260620 20:15:54.829578 5175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:54.830340 5175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:54.831545 5175 hybrid_clock.cc:648] HybridClock initialized: now 1781986554831517 us; error 42 us; skew 500 ppm
Jun 20 20:15:54 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986554, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:54.834978 5175 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:15:54.836023 5175 webserver.cc:460] Webserver started at http://127.24.20.130:42091/ using document root <none> and password file <none>
I20260620 20:15:54.836759 5175 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:54.836871 5175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:15:54 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986554, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:54.860630 5175 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.001s
I20260620 20:15:54.863513 5193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:54.864874 5175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:15:54.864974 5175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "4bdce36f709d460f851b2170b85e96bc"
format_stamp: "Formatted at 2026-06-20 20:15:53 on dist-test-slave-pflr"
server_key: "86516df7d68724e1098379cf2fe2c296"
server_key_iv: "dc34b6305ce4a78ea468ac88021f37ce"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:54.865552 5175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:54.883824 5175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:54.887444 5175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:54.887727 5175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:54.888516 5175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:54.889607 5175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:54.889715 5175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:54.889817 5175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:54.889885 5175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:54.901333 5175 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:37133
I20260620 20:15:54.902287 5175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:15:54.904502 5307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:37133 every 8 connection(s)
I20260620 20:15:54.910967 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 5175
I20260620 20:15:54.911513 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.20.148:43913/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:32849
--local_ip_for_outbound_sockets=127.24.20.131
--tserver_master_addrs=127.24.20.190:34815
--webserver_port=39507
--webserver_interface=127.24.20.131
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.20.148:43913/kms with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:15:54 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986554, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:15:54.927022 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.907003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:51679 (local address 127.24.20.190:34815)
0620 20:15:54.907133 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:54.907137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:54.910491 (+ 3354us) server_negotiation.cc:407] Connection header received
0620 20:15:54.914404 (+ 3913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:54.914407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:54.914468 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:54.914547 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:54.916145 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.917107 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.919014 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.919435 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.919530 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.922903 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:54.922907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:54.922909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:54.922937 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:54.924770 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:54.925349 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:54.925352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:54.925354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:54.925451 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:54.925820 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:54.925824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:54.925825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:54.926048 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:54.926154 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:54.926721 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 20:15:54.926851 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:15:54.928300 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:54.905992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:34815 (local address 127.24.20.130:51679)
0620 20:15:54.910313 (+ 4321us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:54.910369 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:54.914186 (+ 3817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:54.914870 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:54.914882 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:54.915245 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:54.915950 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.915975 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.917284 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.917289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.918841 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:54.918855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:54.919759 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:54.919766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:54.920011 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:54.920792 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:54.920817 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:54.922697 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:54.924886 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:54.924891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:54.924914 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:54.925234 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:54.925550 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:54.925553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:54.925555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:54.925715 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:54.926158 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:54.926164 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:54.926446 (+ 282us) client_negotiation.cc:769] Sending connection context
0620 20:15:54.926646 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 20:15:54.926951 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4087,"thread_start_us":116,"threads_started":1}
I20260620 20:15:54.929562 5308 heartbeater.cc:347] Connected to a master server at 127.24.20.190:34815
I20260620 20:15:54.929785 5308 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:54.930306 5308 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
W20260620 20:15:54.931859 5308 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:34815 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:15:54.931986 5308 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:54.932127 5308 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
I20260620 20:15:54.932839 5308 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:54.932998 5308 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
W20260620 20:15:54.933620 5308 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:15:55.047155 5311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:55.047413 5311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:55.047456 5311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:55.051929 5311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:55.052013 5311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:55.052104 5311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:15:55.059466 5311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.20.148:43913/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:32849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=39507
--tserver_master_addrs=127.24.20.190:34815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:55.060834 5311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:55.062104 5311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:55.073518 5317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:55.073503 5318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:55.073544 5320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:55.073897 5311 server_base.cc:1034] running on GCE node
I20260620 20:15:55.074313 5311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:55.075103 5311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:55.076313 5311 hybrid_clock.cc:648] HybridClock initialized: now 1781986555076279 us; error 42 us; skew 500 ppm
Jun 20 20:15:55 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:55.079932 5311 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:15:55.081060 5311 webserver.cc:460] Webserver started at http://127.24.20.131:39507/ using document root <none> and password file <none>
I20260620 20:15:55.081760 5311 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:55.081872 5311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:15:55 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:55.108007 5311 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 20:15:55.113785 5328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:55.115417 5311 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 20:15:55.115607 5311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "cf66a651366349e783b8ccbf51ee8871"
format_stamp: "Formatted at 2026-06-20 20:15:54 on dist-test-slave-pflr"
server_key: "fe816737e0e0ed47e75ba4dbed17fcb9"
server_key_iv: "7495ea4ee14ed7c193bf69a3cd8245b7"
server_key_version: "kuduclusterkey@0"
I20260620 20:15:55.116503 5311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:55.137925 5311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:55.141728 5311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:55.141990 5311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:55.142768 5311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:15:55.143949 5311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:15:55.144021 5311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:55.144122 5311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:15:55.144176 5311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:55.155076 5311 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:32849
I20260620 20:15:55.155158 5442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:32849 every 8 connection(s)
I20260620 20:15:55.156213 5311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:15:55.157768 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 5311
I20260620 20:15:55.172662 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:55.158579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.24.20.190:34815)
0620 20:15:55.158721 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:55.158728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:55.162086 (+ 3358us) server_negotiation.cc:407] Connection header received
0620 20:15:55.162655 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:55.162664 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:55.162711 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:55.162807 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:55.163571 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:55.164089 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:55.165114 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:55.165326 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:55.165456 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:55.166879 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:55.166883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:55.166886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:55.166918 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:55.169832 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:55.170592 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:55.170599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:55.170602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:55.170689 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:55.171378 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:55.171383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:55.171386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:55.171615 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:55.171807 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:55.172322 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 20:15:55.172452 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 20:15:55 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:15:55.177253 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:55.162404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:47309 (local address 127.24.20.190:34815)
0620 20:15:55.162660 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:55.162670 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:55.163424 (+ 754us) server_negotiation.cc:407] Connection header received
0620 20:15:55.164627 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:55.164630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:55.164682 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:55.164767 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:55.166981 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:55.167770 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:55.169643 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:55.170041 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:55.170114 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:55.173567 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:55.173571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:55.173574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:55.173597 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:55.175158 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:55.175645 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:55.175651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:55.175653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:55.175728 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:55.176065 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:55.176070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:55.176071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:55.176229 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:55.176330 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:55.176940 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 20:15:55.177080 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260620 20:15:55.178203 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:55.162812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:34815 (local address 127.24.20.131:47309)
0620 20:15:55.163285 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:55.163334 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:55.164379 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:55.164938 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:55.164947 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:55.165298 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:55.166758 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:55.166782 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:55.167937 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:55.167943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:55.169501 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:55.169512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:55.170697 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:55.170701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:55.170906 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:55.171450 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:55.171469 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:55.173359 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:55.175271 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:55.175277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:55.175292 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:55.175544 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:55.175824 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:55.175826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:55.175828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:55.175968 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:55.176340 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:55.176348 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:55.176583 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 20:15:55.176874 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 20:15:55.177198 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":133,"threads_started":1}
I20260620 20:15:55.179425 5443 heartbeater.cc:347] Connected to a master server at 127.24.20.190:34815
I20260620 20:15:55.179644 5443 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:55.180173 5443 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
W20260620 20:15:55.181792 5443 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:34815 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:15:55.181934 5443 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:55.182112 5443 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
I20260620 20:15:55.182961 5443 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:55.183128 5443 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
W20260620 20:15:55.183876 5443 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 20:15:55 dist-test-slave-pflr krb5kdc[3975](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781986555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:55.702183 5173 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:55.702504 5173 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
I20260620 20:15:55.703714 4972 ts_manager.cc:194] Registered new tserver with Master: 996f446b7ebd4944a0ff16d2486c1bba (127.24.20.129:37217)
I20260620 20:15:55.705485 4972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:39063
Jun 20 20:15:55 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:55.933835 5308 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:55.934125 5308 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
I20260620 20:15:55.934810 4972 ts_manager.cc:194] Registered new tserver with Master: 4bdce36f709d460f851b2170b85e96bc (127.24.20.130:37133)
I20260620 20:15:55.935566 4972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:51679
Jun 20 20:15:56 dist-test-slave-pflr krb5kdc[3975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986555, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for HTTP/127.24.20.148@KRBTEST.COM
I20260620 20:15:56.184120 5443 heartbeater.cc:464] Registering TS with master...
I20260620 20:15:56.184441 5443 heartbeater.cc:511] Master 127.24.20.190:34815 requested a full tablet report, sending...
I20260620 20:15:56.185214 4972 ts_manager.cc:194] Registered new tserver with Master: cf66a651366349e783b8ccbf51ee8871 (127.24.20.131:32849)
I20260620 20:15:56.185957 4972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:47309
I20260620 20:15:56.191593 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:15:56.226964 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.197062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.24.20.190:34815)
0620 20:15:56.203626 (+ 6564us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:56.203631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:56.203646 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:15:56.203689 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:56.203692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:56.203740 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:56.203823 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:56.204848 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.205687 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.206874 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.207225 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.207324 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.221901 (+ 14577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:56.221907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:56.221910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:56.222049 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:56.224213 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:56.224818 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:56.224822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:56.224824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:56.224911 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:56.225235 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:56.225238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:56.225241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:56.225481 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:56.226567 (+ 1086us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:56.226648 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 20:15:56.226781 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6486,"thread_start_us":84,"threads_started":1}
I20260620 20:15:56.231261 4972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:15:56.236181 4972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:15:56.309682 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.292957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:37217 (local address 127.0.0.1:60932)
0620 20:15:56.297652 (+ 4695us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:56.297705 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:56.297832 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:56.305396 (+ 7564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:56.305399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:56.305414 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:56.305490 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:56.305496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.306791 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.306794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:56.307902 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:56.307912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.309141 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.309144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:56.309246 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.309285 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:15:56.309391 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:15:56.309518 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4500,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260620 20:15:56.310350 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.293084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60932 (local address 127.24.20.129:37217)
0620 20:15:56.304796 (+ 11712us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:56.304803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:56.304827 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:15:56.304903 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:56.304908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:56.305115 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:56.305262 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:56.305618 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.306662 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.308045 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.308815 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.308910 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.309087 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:56.310089 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0620 20:15:56.310210 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11595,"thread_start_us":61,"threads_started":1}
I20260620 20:15:56.311650 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.294808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:37133 (local address 127.0.0.1:51816)
0620 20:15:56.298239 (+ 3431us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:56.298252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:56.298326 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:56.300969 (+ 2643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:56.300971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:56.300985 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:56.301053 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:56.301058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.304485 (+ 3427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.304489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:56.305882 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:56.305891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.310307 (+ 4416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.310311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:56.310631 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.310656 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:15:56.310711 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 20:15:56.311460 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3377,"thread_start_us":74,"threads_started":1}
I20260620 20:15:56.312795 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.295849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:32849 (local address 127.0.0.1:43126)
0620 20:15:56.298476 (+ 2627us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:56.298487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:56.298564 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:56.300682 (+ 2118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:56.300689 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:56.300711 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:56.300848 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:56.300854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.306047 (+ 5193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.306050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:56.308928 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:56.308937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.312418 (+ 3481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.312421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:56.312513 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.312534 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:15:56.312607 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 20:15:56.312680 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2581,"mutex_wait_us":995,"thread_start_us":96,"threads_started":1}
I20260620 20:15:56.313460 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.296232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.24.20.131:32849)
0620 20:15:56.298114 (+ 1882us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:56.298122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:56.298650 (+ 528us) server_negotiation.cc:407] Connection header received
0620 20:15:56.298733 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:56.298738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:56.298977 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:56.299120 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:56.301163 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.302356 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.309055 (+ 6699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.312113 (+ 3058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.312208 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.312359 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:56.313235 (+ 876us) server_negotiation.cc:299] Negotiation successful
0620 20:15:56.313333 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1774,"thread_start_us":95,"threads_started":1}
I20260620 20:15:56.313565 5106 tablet_service.cc:1467] Processing CreateTablet for tablet 4f776cdef8e24de5856e4d02c1ef218e (DEFAULT_TABLE table=test-table [id=2359604ca9284d50b8074ad3207dec2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:56.314721 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f776cdef8e24de5856e4d02c1ef218e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:56.317220 5373 tablet_service.cc:1467] Processing CreateTablet for tablet 4f776cdef8e24de5856e4d02c1ef218e (DEFAULT_TABLE table=test-table [id=2359604ca9284d50b8074ad3207dec2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:56.318280 5373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f776cdef8e24de5856e4d02c1ef218e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:56.324009 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.295168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51816 (local address 127.24.20.130:37133)
0620 20:15:56.300107 (+ 4939us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:56.300114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:56.300137 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:15:56.300210 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:56.300215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:56.300409 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:56.300554 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:56.303253 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.304352 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.307688 (+ 3336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.308521 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.313686 (+ 5165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.313833 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:56.313893 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 20:15:56.313996 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4826,"thread_start_us":80,"threads_started":1}
I20260620 20:15:56.326886 5229 tablet_service.cc:1467] Processing CreateTablet for tablet 4f776cdef8e24de5856e4d02c1ef218e (DEFAULT_TABLE table=test-table [id=2359604ca9284d50b8074ad3207dec2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:15:56.327953 5229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f776cdef8e24de5856e4d02c1ef218e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:56.336138 5480 tablet_bootstrap.cc:492] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871: Bootstrap starting.
I20260620 20:15:56.337517 5481 tablet_bootstrap.cc:492] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc: Bootstrap starting.
I20260620 20:15:56.338471 5480 tablet_bootstrap.cc:654] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:56.339296 5480 log.cc:824] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:56.339493 5481 tablet_bootstrap.cc:654] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:56.340372 5479 tablet_bootstrap.cc:492] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba: Bootstrap starting.
I20260620 20:15:56.341655 5481 log.cc:824] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:56.342471 5479 tablet_bootstrap.cc:654] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:56.343232 5479 log.cc:824] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:56.344403 5481 tablet_bootstrap.cc:492] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc: No bootstrap required, opened a new log
I20260620 20:15:56.344621 5481 ts_tablet_manager.cc:1397] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 20:15:56.345526 5480 tablet_bootstrap.cc:492] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871: No bootstrap required, opened a new log
I20260620 20:15:56.345709 5480 ts_tablet_manager.cc:1397] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 20:15:56.348579 5481 raft_consensus.cc:348] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996f446b7ebd4944a0ff16d2486c1bba" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37217 } } peers { permanent_uuid: "4bdce36f709d460f851b2170b85e96bc" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 37133 } } peers { permanent_uuid: "cf66a651366349e783b8ccbf51ee8871" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32849 } }
I20260620 20:15:56.349133 5481 raft_consensus.cc:374] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:56.349189 5481 raft_consensus.cc:729] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bdce36f709d460f851b2170b85e96bc, State: Initialized, Role: FOLLOWER
I20260620 20:15:56.349422 5480 raft_consensus.cc:348] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996f446b7ebd4944a0ff16d2486c1bba" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37217 } } peers { permanent_uuid: "4bdce36f709d460f851b2170b85e96bc" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 37133 } } peers { permanent_uuid: "cf66a651366349e783b8ccbf51ee8871" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32849 } }
I20260620 20:15:56.349643 5480 raft_consensus.cc:374] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:56.349696 5480 raft_consensus.cc:729] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf66a651366349e783b8ccbf51ee8871, State: Initialized, Role: FOLLOWER
I20260620 20:15:56.349673 5481 consensus_queue.cc:260] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996f446b7ebd4944a0ff16d2486c1bba" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37217 } } peers { permanent_uuid: "4bdce36f709d460f851b2170b85e96bc" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 37133 } } peers { permanent_uuid: "cf66a651366349e783b8ccbf51ee8871" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32849 } }
I20260620 20:15:56.350167 5480 consensus_queue.cc:260] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996f446b7ebd4944a0ff16d2486c1bba" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37217 } } peers { permanent_uuid: "4bdce36f709d460f851b2170b85e96bc" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 37133 } } peers { permanent_uuid: "cf66a651366349e783b8ccbf51ee8871" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32849 } }
I20260620 20:15:56.351035 5480 ts_tablet_manager.cc:1428] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260620 20:15:56.351544 5481 ts_tablet_manager.cc:1428] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260620 20:15:56.351869 5308 heartbeater.cc:503] Master 127.24.20.190:34815 was elected leader, sending a full tablet report...
I20260620 20:15:56.353348 5479 tablet_bootstrap.cc:492] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba: No bootstrap required, opened a new log
I20260620 20:15:56.353528 5479 ts_tablet_manager.cc:1397] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260620 20:15:56.357237 5479 raft_consensus.cc:348] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996f446b7ebd4944a0ff16d2486c1bba" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37217 } } peers { permanent_uuid: "4bdce36f709d460f851b2170b85e96bc" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 37133 } } peers { permanent_uuid: "cf66a651366349e783b8ccbf51ee8871" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32849 } }
I20260620 20:15:56.357465 5479 raft_consensus.cc:374] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:56.357532 5479 raft_consensus.cc:729] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996f446b7ebd4944a0ff16d2486c1bba, State: Initialized, Role: FOLLOWER
I20260620 20:15:56.357939 5479 consensus_queue.cc:260] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996f446b7ebd4944a0ff16d2486c1bba" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37217 } } peers { permanent_uuid: "4bdce36f709d460f851b2170b85e96bc" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 37133 } } peers { permanent_uuid: "cf66a651366349e783b8ccbf51ee8871" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32849 } }
I20260620 20:15:56.358525 5173 heartbeater.cc:503] Master 127.24.20.190:34815 was elected leader, sending a full tablet report...
I20260620 20:15:56.358846 5479 ts_tablet_manager.cc:1428] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260620 20:15:56.359043 5443 heartbeater.cc:503] Master 127.24.20.190:34815 was elected leader, sending a full tablet report...
W20260620 20:15:56.404621 5309 tablet.cc:2369] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:15:56.409215 5444 tablet.cc:2369] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:15:56.411191 5174 tablet.cc:2369] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:15:56.698921 5487 raft_consensus.cc:482] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:15:56.699210 5487 raft_consensus.cc:504] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996f446b7ebd4944a0ff16d2486c1bba" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37217 } } peers { permanent_uuid: "4bdce36f709d460f851b2170b85e96bc" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 37133 } } peers { permanent_uuid: "cf66a651366349e783b8ccbf51ee8871" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32849 } }
I20260620 20:15:56.700451 5487 leader_election.cc:288] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bdce36f709d460f851b2170b85e96bc (127.24.20.130:37133), cf66a651366349e783b8ccbf51ee8871 (127.24.20.131:32849)
I20260620 20:15:56.704516 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.700807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:55061 (local address 127.24.20.131:32849)
0620 20:15:56.700957 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:56.700962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:56.701304 (+ 342us) server_negotiation.cc:407] Connection header received
0620 20:15:56.701441 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:56.701444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:56.701493 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:56.701594 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:56.701853 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.702335 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.703583 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.703953 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.704080 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.704141 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:56.704309 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 20:15:56.704381 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:15:56.704545 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.700719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:37133 (local address 127.24.20.129:35491)
0620 20:15:56.701094 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:56.701107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:56.701209 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:56.701451 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:56.701454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:56.701469 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:56.701595 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:56.701602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.702537 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.702540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:56.703377 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:56.703385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.704137 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.704140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:56.704225 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.704241 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 20:15:56.704336 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 20:15:56.704397 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":1,"thread_start_us":123,"threads_started":1}
I20260620 20:15:56.704571 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.700802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:35491 (local address 127.24.20.130:37133)
0620 20:15:56.700957 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:56.700962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:56.701210 (+ 248us) server_negotiation.cc:407] Connection header received
0620 20:15:56.701290 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:56.701293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:56.701349 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:56.701438 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:15:56.701719 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.702342 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.703512 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.703966 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.704112 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.704163 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:56.704376 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 20:15:56.704454 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:15:56.704557 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.700719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:32849 (local address 127.24.20.129:55061)
0620 20:15:56.701149 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:56.701189 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:56.701316 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:56.701604 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:56.701607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:56.701621 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:15:56.701737 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:56.701744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.702466 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.702469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:56.703416 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:56.703426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.704078 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.704083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:56.704168 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.704199 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 20:15:56.704275 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 20:15:56.704368 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":64,"thread_start_us":111,"threads_started":1}
I20260620 20:15:56.705199 5260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f776cdef8e24de5856e4d02c1ef218e" candidate_uuid: "996f446b7ebd4944a0ff16d2486c1bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bdce36f709d460f851b2170b85e96bc" is_pre_election: true
I20260620 20:15:56.705200 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f776cdef8e24de5856e4d02c1ef218e" candidate_uuid: "996f446b7ebd4944a0ff16d2486c1bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf66a651366349e783b8ccbf51ee8871" is_pre_election: true
I20260620 20:15:56.705528 5384 raft_consensus.cc:2454] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 996f446b7ebd4944a0ff16d2486c1bba in term 0.
I20260620 20:15:56.705528 5260 raft_consensus.cc:2454] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 996f446b7ebd4944a0ff16d2486c1bba in term 0.
I20260620 20:15:56.706051 5061 leader_election.cc:302] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996f446b7ebd4944a0ff16d2486c1bba, cf66a651366349e783b8ccbf51ee8871; no voters:
I20260620 20:15:56.706350 5487 raft_consensus.cc:2781] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:15:56.706430 5487 raft_consensus.cc:482] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:15:56.706517 5487 raft_consensus.cc:3037] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:56.707612 5487 raft_consensus.cc:504] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996f446b7ebd4944a0ff16d2486c1bba" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37217 } } peers { permanent_uuid: "4bdce36f709d460f851b2170b85e96bc" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 37133 } } peers { permanent_uuid: "cf66a651366349e783b8ccbf51ee8871" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32849 } }
I20260620 20:15:56.707979 5487 leader_election.cc:288] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [CANDIDATE]: Term 1 election: Requested vote from peers 4bdce36f709d460f851b2170b85e96bc (127.24.20.130:37133), cf66a651366349e783b8ccbf51ee8871 (127.24.20.131:32849)
I20260620 20:15:56.708335 5260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f776cdef8e24de5856e4d02c1ef218e" candidate_uuid: "996f446b7ebd4944a0ff16d2486c1bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bdce36f709d460f851b2170b85e96bc"
I20260620 20:15:56.708448 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f776cdef8e24de5856e4d02c1ef218e" candidate_uuid: "996f446b7ebd4944a0ff16d2486c1bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf66a651366349e783b8ccbf51ee8871"
I20260620 20:15:56.708482 5260 raft_consensus.cc:3037] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:56.708570 5384 raft_consensus.cc:3037] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:56.709801 5260 raft_consensus.cc:2454] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 996f446b7ebd4944a0ff16d2486c1bba in term 1.
I20260620 20:15:56.709801 5384 raft_consensus.cc:2454] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 996f446b7ebd4944a0ff16d2486c1bba in term 1.
I20260620 20:15:56.710134 5061 leader_election.cc:302] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996f446b7ebd4944a0ff16d2486c1bba, cf66a651366349e783b8ccbf51ee8871; no voters:
I20260620 20:15:56.710382 5487 raft_consensus.cc:2781] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:56.710646 5487 raft_consensus.cc:686] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [term 1 LEADER]: Becoming Leader. State: Replica: 996f446b7ebd4944a0ff16d2486c1bba, State: Running, Role: LEADER
I20260620 20:15:56.710999 5487 consensus_queue.cc:237] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996f446b7ebd4944a0ff16d2486c1bba" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37217 } } peers { permanent_uuid: "4bdce36f709d460f851b2170b85e96bc" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 37133 } } peers { permanent_uuid: "cf66a651366349e783b8ccbf51ee8871" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32849 } }
I20260620 20:15:56.714040 4971 catalog_manager.cc:5495] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba reported cstate change: term changed from 0 to 1, leader changed from <none> to 996f446b7ebd4944a0ff16d2486c1bba (127.24.20.129). New cstate: current_term: 1 leader_uuid: "996f446b7ebd4944a0ff16d2486c1bba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "996f446b7ebd4944a0ff16d2486c1bba" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bdce36f709d460f851b2170b85e96bc" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 37133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf66a651366349e783b8ccbf51ee8871" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32849 } health_report { overall_health: UNKNOWN } } }
I20260620 20:15:56.760499 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:56.756587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.24.20.129:37217)
0620 20:15:56.756745 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:56.756749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:56.756888 (+ 139us) server_negotiation.cc:407] Connection header received
0620 20:15:56.757103 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:56.757106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:56.757162 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:56.757279 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:15:56.757536 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.758087 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.759035 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:56.759259 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:56.759372 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:56.759690 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:15:56.760122 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:15:56.760262 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:56.760302 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 20:15:56.760356 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:15:56.766520 5384 raft_consensus.cc:1261] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871 [term 1 FOLLOWER]: Refusing update from remote peer 996f446b7ebd4944a0ff16d2486c1bba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:56.766533 5260 raft_consensus.cc:1261] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc [term 1 FOLLOWER]: Refusing update from remote peer 996f446b7ebd4944a0ff16d2486c1bba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:56.767186 5487 consensus_queue.cc:1035] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf66a651366349e783b8ccbf51ee8871" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 32849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:56.767405 5497 consensus_queue.cc:1035] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bdce36f709d460f851b2170b85e96bc" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 37133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:56.783439 4972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:56.783634 4972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:15:56.785868 4972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a41d7e966c8c423cbe63e26dca88b12e: Sending DeleteTablet for 3 replicas of tablet 4f776cdef8e24de5856e4d02c1ef218e
I20260620 20:15:56.786625 5373 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f776cdef8e24de5856e4d02c1ef218e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:56 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:43126
I20260620 20:15:56.786593 5229 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f776cdef8e24de5856e4d02c1ef218e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:56 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:51816
I20260620 20:15:56.786621 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f776cdef8e24de5856e4d02c1ef218e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:15:56 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:60932
I20260620 20:15:56.787138 5510 tablet_replica.cc:330] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba: stopping tablet replica
I20260620 20:15:56.787457 5510 raft_consensus.cc:2229] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:15:56.787719 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 5010
I20260620 20:15:56.787810 5510 raft_consensus.cc:2258] T 4f776cdef8e24de5856e4d02c1ef218e P 996f446b7ebd4944a0ff16d2486c1bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:56.789606 5511 tablet_replica.cc:330] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871: stopping tablet replica
I20260620 20:15:56.789882 5511 raft_consensus.cc:2229] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:56.790167 5511 raft_consensus.cc:2258] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:56.790529 5512 tablet_replica.cc:330] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc: stopping tablet replica
I20260620 20:15:56.790807 5512 raft_consensus.cc:2229] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:15:56.791069 5512 raft_consensus.cc:2258] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:15:56.791756 5511 ts_tablet_manager.cc:1905] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:56.792279 5512 ts_tablet_manager.cc:1905] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:15:56.794384 5512 ts_tablet_manager.cc:1918] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:15:56.794495 5512 log.cc:1196] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/4f776cdef8e24de5856e4d02c1ef218e
I20260620 20:15:56.794795 5512 ts_tablet_manager.cc:1939] T 4f776cdef8e24de5856e4d02c1ef218e P 4bdce36f709d460f851b2170b85e96bc: Deleting consensus metadata
I20260620 20:15:56.795204 5511 ts_tablet_manager.cc:1918] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:15:56.795301 5511 log.cc:1196] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/4f776cdef8e24de5856e4d02c1ef218e
I20260620 20:15:56.795626 5511 ts_tablet_manager.cc:1939] T 4f776cdef8e24de5856e4d02c1ef218e P cf66a651366349e783b8ccbf51ee8871: Deleting consensus metadata
I20260620 20:15:56.795907 4958 catalog_manager.cc:4841] TS 4bdce36f709d460f851b2170b85e96bc (127.24.20.130:37133): tablet 4f776cdef8e24de5856e4d02c1ef218e (table test-table [id=2359604ca9284d50b8074ad3207dec2b]) successfully deleted
W20260620 20:15:56.796366 4957 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:37217 (error 108)
I20260620 20:15:56.796797 4957 catalog_manager.cc:4841] TS cf66a651366349e783b8ccbf51ee8871 (127.24.20.131:32849): tablet 4f776cdef8e24de5856e4d02c1ef218e (table test-table [id=2359604ca9284d50b8074ad3207dec2b]) successfully deleted
W20260620 20:15:56.797423 5475 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:15:56.797075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:37217 (local address 127.0.0.1:57172)
0620 20:15:56.797182 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:56.797196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:56.797276 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:56.797347 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:37217: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 20:15:56.797588 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 5175
W20260620 20:15:56.797647 4957 catalog_manager.cc:4584] TS 996f446b7ebd4944a0ff16d2486c1bba (127.24.20.129:37217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f776cdef8e24de5856e4d02c1ef218e: Network error: Client connection negotiation failed: client connection to 127.24.20.129:37217: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260620 20:15:56.803324 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 5311
I20260620 20:15:56.808362 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 4938
I20260620 20:15:56.814510 24658 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 20:15:57.159071 24658 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 20:15:57.159209 24658 mini_ranger.cc:66] Stopping Ranger...
I20260620 20:15:57.503696 24658 mini_ranger.cc:68] Stopped Ranger
2026-06-20 20:15:57.503 UTC [4005] LOG: received smart shutdown request
2026-06-20 20:15:57.508 UTC [4005] LOG: background worker "logical replication launcher" (PID 4014) exited with exit code 1
2026-06-20 20:15:57.508 UTC [4009] LOG: shutting down
2026-06-20 20:15:57.677 UTC [4005] LOG: database system is shut down
2026-06-20T20:15:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48088 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:15:57 dist-test-slave-pflr krb5kdc[5522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:15:57 dist-test-slave-pflr krb5kdc[5522](info): set up 2 sockets
Jun 20 20:15:57 dist-test-slave-pflr krb5kdc[5522](info): commencing operation
krb5kdc: starting...
W20260620 20:15:58.845078 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:15:58 dist-test-slave-pflr krb5kdc[5522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:15:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:15:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:15:58.962033 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:33311
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:46841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:33311,127.24.20.189:33005,127.24.20.188:36349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:15:59.068516 5535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:59.068879 5535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:59.068959 5535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:59.072841 5535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:59.072926 5535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:59.072963 5535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:59.072999 5535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:59.073040 5535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:59.078210 5535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:46841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:33311,127.24.20.189:33005,127.24.20.188:36349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:33311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:59.079244 5535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:59.080775 5535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:59.090606 5544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:59.090593 5541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:59.090615 5542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:59.090823 5535 server_base.cc:1034] running on GCE node
I20260620 20:15:59.091449 5535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:59.092489 5535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:59.093679 5535 hybrid_clock.cc:648] HybridClock initialized: now 1781986559093668 us; error 28 us; skew 500 ppm
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:59.096292 5535 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:15:59.097177 5535 webserver.cc:460] Webserver started at http://127.24.20.190:41183/ using document root <none> and password file <none>
I20260620 20:15:59.097723 5535 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:59.097808 5535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:59.098038 5535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:59.099984 5535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "d60f83cc9f44428ea3d458f3ce98bcd5"
format_stamp: "Formatted at 2026-06-20 20:15:59 on dist-test-slave-pflr"
server_key: "e53f761c3166d4ebe4d2564ce20ff918"
server_key_iv: "ac48bc412f3245ecb091d953666cb276"
server_key_version: "encryptionkey@0"
I20260620 20:15:59.100557 5535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "d60f83cc9f44428ea3d458f3ce98bcd5"
format_stamp: "Formatted at 2026-06-20 20:15:59 on dist-test-slave-pflr"
server_key: "e53f761c3166d4ebe4d2564ce20ff918"
server_key_iv: "ac48bc412f3245ecb091d953666cb276"
server_key_version: "encryptionkey@0"
I20260620 20:15:59.104014 5535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:15:59.106359 5551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:59.107481 5535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 20:15:59.107616 5535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "d60f83cc9f44428ea3d458f3ce98bcd5"
format_stamp: "Formatted at 2026-06-20 20:15:59 on dist-test-slave-pflr"
server_key: "e53f761c3166d4ebe4d2564ce20ff918"
server_key_iv: "ac48bc412f3245ecb091d953666cb276"
server_key_version: "encryptionkey@0"
I20260620 20:15:59.107718 5535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:59.126116 5535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:59.128903 5535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:59.129096 5535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:59.135874 5535 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:33311
I20260620 20:15:59.135891 5603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:33311 every 8 connection(s)
I20260620 20:15:59.136838 5535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:15:59.137621 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 5535
I20260620 20:15:59.137751 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:15:59.138063 24658 external_mini_cluster.cc:1428] Setting key cf155c361b4cfec1cef87c66c825d332
I20260620 20:15:59.140127 5604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:59.144719 5604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.20.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.189@KRBTEST.COM" created.
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:15:59.160276 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.147043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33311 (local address 127.0.0.1:34680)
0620 20:15:59.147655 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.147705 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:59.148688 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:59.149484 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:59.149489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:59.149925 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:59.150154 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.150167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.151332 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.151334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.152076 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.152085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.153022 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.153025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.153193 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.153798 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:59.153814 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:59.155211 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:59.157526 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.157534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.157541 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.157755 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.158143 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.158146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.158147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.158256 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.158916 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:59.158921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:59.159039 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 20:15:59.159194 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 20:15:59.159550 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":139,"thread_start_us":171,"threads_started":1}
I20260620 20:15:59.160282 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.147945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.24.20.190:33311)
0620 20:15:59.148232 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.148242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.148316 (+ 74us) server_negotiation.cc:407] Connection header received
0620 20:15:59.149030 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.149035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.149258 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.149497 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.150305 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.151169 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.152199 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.152872 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.153139 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.155362 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.155380 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.155397 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.155422 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.157415 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.157875 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.157882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.157884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.158023 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.158354 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.158357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.158358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.158798 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.158910 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.159266 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.159475 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
I20260620 20:15:59.162768 5604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:15:59.163630 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.163277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:33005 (local address 127.0.0.1:53594)
0620 20:15:59.163388 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.163516 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 20:15:59.163853 5553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.20.189:33005: connect: Connection refused (error 111)
I20260620 20:15:59.164743 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.164454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:33005 (local address 127.0.0.1:53608)
0620 20:15:59.164633 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.164684 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 20:15:59.164891 5604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.189:33005: Network error: Client connection negotiation failed: client connection to 127.24.20.189:33005: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.189.keytab.
Entry for principal kudu/127.24.20.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.189.keytab.
I20260620 20:15:59.167505 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.189:33005
--webserver_interface=127.24.20.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.189.keytab
--principal=kudu/127.24.20.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:46841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:33311,127.24.20.189:33005,127.24.20.188:36349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:15:59.214135 5604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } attempt: 1
I20260620 20:15:59.214993 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.214698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:33005 (local address 127.0.0.1:53618)
0620 20:15:59.214849 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.214914 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 20:15:59.215791 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.215605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:33005 (local address 127.0.0.1:53628)
0620 20:15:59.215697 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.215739 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 20:15:59.215934 5604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.189:33005: Network error: Client connection negotiation failed: client connection to 127.24.20.189:33005: connect: Connection refused (error 111)
W20260620 20:15:59.275422 5611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:59.275696 5611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:59.275766 5611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:59.279799 5611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:59.279876 5611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:59.279915 5611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:59.279973 5611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:59.280012 5611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:59.284096 5604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } attempt: 2
I20260620 20:15:59.284987 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.284706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:33005 (local address 127.0.0.1:53640)
0620 20:15:59.284847 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.284912 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 20:15:59.285452 5611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:46841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:33311,127.24.20.189:33005,127.24.20.188:36349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.189.keytab
--principal=kudu/127.24.20.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.189:33005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.20.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:59.285696 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.285474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:33005 (local address 127.0.0.1:53644)
0620 20:15:59.285589 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.285640 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 20:15:59.285845 5604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.189:33005: Network error: Client connection negotiation failed: client connection to 127.24.20.189:33005: connect: Connection refused (error 111)
I20260620 20:15:59.286521 5611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:59.287449 5611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:15:59.295418 5619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:59.295432 5617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:59.295596 5611 server_base.cc:1034] running on GCE node
W20260620 20:15:59.295450 5616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:59.296124 5611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:59.297200 5611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:59.298357 5611 hybrid_clock.cc:648] HybridClock initialized: now 1781986559298346 us; error 24 us; skew 500 ppm
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:59.301057 5611 init.cc:377] Logged in from keytab as kudu/127.24.20.189@KRBTEST.COM (short username kudu)
I20260620 20:15:59.301889 5611 webserver.cc:460] Webserver started at http://127.24.20.189:38323/ using document root <none> and password file <none>
I20260620 20:15:59.302438 5611 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:59.302498 5611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:59.302731 5611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:59.304752 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/data/instance:
uuid: "a4e62fcd6bf74404ae9f70710cac0066"
format_stamp: "Formatted at 2026-06-20 20:15:59 on dist-test-slave-pflr"
server_key: "e170129d7955cbceb4f07f5fff287ad9"
server_key_iv: "75b21bf4cf017446cc0e91f014107659"
server_key_version: "encryptionkey@0"
I20260620 20:15:59.305290 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/wal/instance:
uuid: "a4e62fcd6bf74404ae9f70710cac0066"
format_stamp: "Formatted at 2026-06-20 20:15:59 on dist-test-slave-pflr"
server_key: "e170129d7955cbceb4f07f5fff287ad9"
server_key_iv: "75b21bf4cf017446cc0e91f014107659"
server_key_version: "encryptionkey@0"
I20260620 20:15:59.308773 5611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:15:59.311081 5626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:59.312407 5611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:15:59.312558 5611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/wal
uuid: "a4e62fcd6bf74404ae9f70710cac0066"
format_stamp: "Formatted at 2026-06-20 20:15:59 on dist-test-slave-pflr"
server_key: "e170129d7955cbceb4f07f5fff287ad9"
server_key_iv: "75b21bf4cf017446cc0e91f014107659"
server_key_version: "encryptionkey@0"
I20260620 20:15:59.312729 5611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:59.324012 5611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:59.326778 5611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:59.326965 5611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:59.333591 5611 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.189:33005
I20260620 20:15:59.333601 5678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.189:33005 every 8 connection(s)
I20260620 20:15:59.334445 5611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/data/info.pb
I20260620 20:15:59.337059 5679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:59.340463 5679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:15:59.343523 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 5611
I20260620 20:15:59.343668 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-1/wal/instance
I20260620 20:15:59.343933 24658 external_mini_cluster.cc:1428] Setting key cb5a38b7537fe1e49eda5575d50250f3
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.189@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.20.188@KRBTEST.COM; defaulting to no policy
I20260620 20:15:59.353459 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.342204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.24.20.190:33311)
0620 20:15:59.342332 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.342336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.343086 (+ 750us) server_negotiation.cc:407] Connection header received
0620 20:15:59.344028 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.344039 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.344097 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.344189 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.345327 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.345866 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.347009 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.347265 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.347406 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.349755 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.349764 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.349766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.349788 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.351237 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.351771 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.351776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.351778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.351854 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.352195 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.352200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.352201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.352469 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.352587 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.353111 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.353264 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.188@KRBTEST.COM" created.
I20260620 20:15:59.354280 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.342545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33311 (local address 127.0.0.1:34688)
0620 20:15:59.342933 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.342973 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:59.343833 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:59.344346 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:59.344353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:59.344737 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:59.345180 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.345193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.345997 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.346002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.346878 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.346888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.347438 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.347443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.347615 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.348177 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:59.348192 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:59.349594 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:59.351359 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.351364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.351377 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.351667 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.351954 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.351957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.351959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.352095 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.352610 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:59.352618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:59.352861 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 20:15:59.353047 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 20:15:59.353337 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
I20260620 20:15:59.355777 5679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.189@KRBTEST.COM for kudu/127.24.20.189@KRBTEST.COM
I20260620 20:15:59.364960 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.356246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:33005 (local address 127.0.0.1:53646)
0620 20:15:59.356353 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.356368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:59.356439 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:59.357056 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:59.357059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:59.357217 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:59.357301 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.357306 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.358286 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.358292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.358834 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.358841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.359379 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.359382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.359470 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.359913 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:59.359926 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:59.361342 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:59.363172 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.363175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.363177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.363497 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.363875 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.363877 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.363879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.363956 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.364435 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:59.364437 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:59.364511 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 20:15:59.364662 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 20:15:59.364793 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 20:15:59.364985 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.356379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.24.20.189:33005)
0620 20:15:59.356736 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.356753 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.356771 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:15:59.356803 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.356806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.356939 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.357084 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.357404 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.358173 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.358943 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.359272 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.359382 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.361471 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.361478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.361487 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.361507 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.363062 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.363645 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.363654 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.363658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.363759 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.364039 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.364041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.364043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.364338 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.364442 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.364753 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.364860 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":136,"threads_started":1}
I20260620 20:15:59.366777 5679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:15:59.367475 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.367205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59772)
0620 20:15:59.367305 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.367387 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 20:15:59.367654 5628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
I20260620 20:15:59.368489 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.368231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59782)
0620 20:15:59.368348 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.368429 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 20:15:59.368644 5679 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:36349: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.188.keytab.
Entry for principal kudu/127.24.20.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.188.keytab.
I20260620 20:15:59.369730 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.188:36349
--webserver_interface=127.24.20.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.188.keytab
--principal=kudu/127.24.20.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:46841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:33311,127.24.20.189:33005,127.24.20.188:36349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:15:59.377010 5604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } attempt: 3
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.189@KRBTEST.COM
I20260620 20:15:59.385730 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.377531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.24.20.189:33005)
0620 20:15:59.377653 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.377659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.377713 (+ 54us) server_negotiation.cc:407] Connection header received
0620 20:15:59.377874 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.377877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.377936 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.378022 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.378442 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.378953 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.380138 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.380460 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.380583 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.382763 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.382766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.382768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.382788 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.384101 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.384565 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.384567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.384569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.384727 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.385037 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.385039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.385040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.385191 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.385263 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.385528 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.385629 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:15:59.385771 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.377482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:33005 (local address 127.0.0.1:53654)
0620 20:15:59.377628 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.377643 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:59.377757 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:59.378043 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:59.378046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:59.378225 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:59.378326 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.378332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.379072 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.379076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.380003 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.380012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.380574 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.380578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.380711 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.381213 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:59.381224 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:59.382640 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:59.384205 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.384208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.384210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.384456 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.384825 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.384831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.384835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.384938 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.385279 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:59.385282 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:59.385380 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 20:15:59.385484 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:15:59.385608 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 20:15:59.386974 5604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:15:59.387784 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.387494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59790)
0620 20:15:59.387614 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.387668 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 20:15:59.388522 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.388276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59794)
0620 20:15:59.388433 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.388475 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260620 20:15:59.388746 5604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:36349: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
I20260620 20:15:59.417855 5679 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } attempt: 1
I20260620 20:15:59.418654 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.418378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59798)
0620 20:15:59.418517 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.418579 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 20:15:59.419348 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.419145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59804)
0620 20:15:59.419249 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.419300 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 20:15:59.419471 5679 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:36349: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
I20260620 20:15:59.419901 5604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } attempt: 1
I20260620 20:15:59.420514 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.420292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59810)
0620 20:15:59.420408 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.420457 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:15:59.421298 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.421071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59814)
0620 20:15:59.421195 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.421243 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 20:15:59.421423 5604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:36349: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
W20260620 20:15:59.481226 5686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:15:59.481496 5686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:15:59.481576 5686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:15:59.485594 5686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:15:59.485663 5686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:15:59.485702 5686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:15:59.485759 5686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:15:59.485798 5686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:15:59.487637 5679 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } attempt: 2
I20260620 20:15:59.488432 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.488154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59820)
0620 20:15:59.488291 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.488360 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 20:15:59.489118 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.488897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59828)
0620 20:15:59.489004 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.489068 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 20:15:59.489231 5679 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:36349: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
I20260620 20:15:59.491220 5686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:46841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:33311,127.24.20.189:33005,127.24.20.188:36349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.188.keytab
--principal=kudu/127.24.20.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.188:36349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.20.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:15:59.492195 5686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:15:59.493279 5686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:15:59.496582 5604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } attempt: 2
I20260620 20:15:59.497380 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.497088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59840)
0620 20:15:59.497237 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.497298 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 20:15:59.498106 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.497868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59848)
0620 20:15:59.497976 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.498029 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 20:15:59.498296 5604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:36349: Network error: Client connection negotiation failed: client connection to 127.24.20.188:36349: connect: Connection refused (error 111)
W20260620 20:15:59.501711 5691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:59.501688 5694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:15:59.501726 5692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:15:59.502024 5686 server_base.cc:1034] running on GCE node
I20260620 20:15:59.502609 5686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:15:59.503679 5686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:15:59.504858 5686 hybrid_clock.cc:648] HybridClock initialized: now 1781986559504827 us; error 50 us; skew 500 ppm
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:15:59.507490 5686 init.cc:377] Logged in from keytab as kudu/127.24.20.188@KRBTEST.COM (short username kudu)
I20260620 20:15:59.508311 5686 webserver.cc:460] Webserver started at http://127.24.20.188:41633/ using document root <none> and password file <none>
I20260620 20:15:59.508915 5686 fs_manager.cc:359] Metadata directory not provided
I20260620 20:15:59.509004 5686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:15:59.509231 5686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:15:59.511292 5686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/data/instance:
uuid: "3a35728eff2a4b7580c39f7673d80f39"
format_stamp: "Formatted at 2026-06-20 20:15:59 on dist-test-slave-pflr"
server_key: "7e03485926d9c1fbf86ab2b69940273d"
server_key_iv: "276ec4470b4f3a0db8f9826330164875"
server_key_version: "encryptionkey@0"
I20260620 20:15:59.511940 5686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/wal/instance:
uuid: "3a35728eff2a4b7580c39f7673d80f39"
format_stamp: "Formatted at 2026-06-20 20:15:59 on dist-test-slave-pflr"
server_key: "7e03485926d9c1fbf86ab2b69940273d"
server_key_iv: "276ec4470b4f3a0db8f9826330164875"
server_key_version: "encryptionkey@0"
I20260620 20:15:59.515514 5686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 20:15:59.518041 5701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:15:59.519155 5686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:15:59.519276 5686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/wal
uuid: "3a35728eff2a4b7580c39f7673d80f39"
format_stamp: "Formatted at 2026-06-20 20:15:59 on dist-test-slave-pflr"
server_key: "7e03485926d9c1fbf86ab2b69940273d"
server_key_iv: "276ec4470b4f3a0db8f9826330164875"
server_key_version: "encryptionkey@0"
I20260620 20:15:59.519377 5686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:15:59.534076 5686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:15:59.537101 5686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:15:59.537297 5686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:15:59.544298 5686 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.188:36349
I20260620 20:15:59.544373 5754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.188:36349 every 8 connection(s)
I20260620 20:15:59.545328 5686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/data/info.pb
I20260620 20:15:59.548229 5755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:15:59.551839 5755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:15:59.555576 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 5686
I20260620 20:15:59.555717 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986405628821-24658-0/minicluster-data/master-2/wal/instance
I20260620 20:15:59.556008 24658 external_mini_cluster.cc:1428] Setting key 542962730cf3ebd1d240989cb36a0d17
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.188@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.188@KRBTEST.COM
I20260620 20:15:59.566144 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.553688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.24.20.190:33311)
0620 20:15:59.553830 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.553833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.554623 (+ 790us) server_negotiation.cc:407] Connection header received
0620 20:15:59.555453 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.555460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.555521 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.555622 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.557101 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.557799 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.559314 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.559669 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.559748 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.562357 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.562361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.562364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.562406 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.564101 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.564615 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.564621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.564624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.564704 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.565034 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.565038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.565040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.565237 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.565342 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.565857 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.565970 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:15:59.566906 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.554026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33311 (local address 127.0.0.1:34694)
0620 20:15:59.554453 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.554493 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:59.555226 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:59.555838 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:59.555844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:59.556230 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:59.556923 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.556945 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.557955 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.557961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.559200 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.559209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.559808 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.559812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.560003 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.560634 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:59.560661 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:59.562216 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:59.564202 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.564213 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.564222 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.564471 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.564797 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.564801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.564803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.564934 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.565352 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:59.565360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:59.565621 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 20:15:59.565805 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 20:15:59.566089 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":96,"threads_started":1}
I20260620 20:15:59.567826 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.556680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.24.20.188:36349)
0620 20:15:59.557069 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.557079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.557106 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:15:59.557224 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.557228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.557367 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.557487 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.558417 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.559478 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.560584 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.560971 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.561060 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.563754 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.563761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.563776 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.563805 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.565582 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.566034 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.566044 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.566045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.566116 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.566547 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.566551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.566553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.566771 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.566991 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.567507 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.567642 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":160,"threads_started":1}
I20260620 20:15:59.568437 5755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.188@KRBTEST.COM for kudu/127.24.20.189@KRBTEST.COM
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:15:59.576730 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.569001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:33005 (local address 127.0.0.1:53662)
0620 20:15:59.569119 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.569134 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:59.569205 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:59.569573 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:59.569576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:59.569711 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:59.569789 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.569818 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.570617 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.570620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.571109 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.571117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.571557 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.571561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.571655 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.572085 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:59.572104 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:59.573383 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:59.575117 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.575120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.575122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.575398 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.575729 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.575731 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.575732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.575843 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.576315 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:59.576318 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:59.576387 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 20:15:59.576469 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 20:15:59.576561 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 20:15:59.576941 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.569114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.24.20.189:33005)
0620 20:15:59.569247 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.569250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.569263 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:15:59.569301 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.569303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.569367 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.569447 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.569998 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.570483 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.571233 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.571446 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.571550 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.573520 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.573523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.573526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.573551 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.574980 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.575508 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.575514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.575516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.575618 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.575951 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.575954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.575956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.576209 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.576330 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.576580 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.576742 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:15:59.577658 5755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:15:59.578642 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.569972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.24.20.190:33311)
0620 20:15:59.570085 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.570088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.570101 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:15:59.570199 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.570202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.570237 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.570327 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.570918 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.571386 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.572134 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.572368 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.572476 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.574977 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.574981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.574984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.575022 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.576876 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.577353 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.577356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.577358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.577451 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.577732 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.577736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.577738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.577942 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.578167 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.578361 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.578467 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:15:59.580376 5679 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } attempt: 3
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.188@KRBTEST.COM for kudu/127.24.20.188@KRBTEST.COM
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.189@KRBTEST.COM
I20260620 20:15:59.587034 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.578129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59860)
0620 20:15:59.578358 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.578374 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:59.578449 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:59.578673 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:59.578676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:59.578847 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:59.578922 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.578927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.579690 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.579693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.580777 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.580789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.581290 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.581294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.581376 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.581850 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:59.581863 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:59.583307 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:59.585355 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.585358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.585360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.585567 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.585848 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.585853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.585859 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.585953 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.586497 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:59.586499 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:59.586587 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 20:15:59.586698 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:15:59.586849 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.189@KRBTEST.COM for kudu/127.24.20.188@KRBTEST.COM
I20260620 20:15:59.587569 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.578355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59860 (local address 127.24.20.188:36349)
0620 20:15:59.578462 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.578468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.578495 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:15:59.578529 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.578532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.578575 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.578662 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.579037 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.579571 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.580917 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.581203 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.581282 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.583461 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.583466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.583470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.583494 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.585265 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.585660 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.585663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.585664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.585740 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.586085 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.586087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.586089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.586326 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.586453 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.587250 (+ 797us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.587367 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:15:59.589737 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.579714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53664 (local address 127.24.20.189:33005)
0620 20:15:59.579829 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.579836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.579850 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:15:59.579981 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.579983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.580019 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.580089 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.580934 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.581629 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.582534 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.582834 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.582933 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.585615 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.585619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.585622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.585653 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.587399 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.588333 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.588337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.588339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.588431 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.588783 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.588789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.588794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.588997 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.589221 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.589381 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.589565 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 20:15:59.590996 5755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39: Bootstrap starting.
I20260620 20:15:59.591837 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.580765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59876)
0620 20:15:59.581028 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.581042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:59.581122 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:59.582176 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:59.582179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:59.582355 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:59.582457 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.582463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.584244 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.584249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.585132 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.585145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.585793 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.585800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.585900 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.586352 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:59.586372 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:59.587729 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:59.590010 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.590013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.590015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.590248 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.590791 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.590794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.590796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.590879 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.591339 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:59.591342 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:59.591444 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 20:15:59.591563 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 20:15:59.591698 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260620 20:15:59.592327 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.580806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59876 (local address 127.24.20.188:36349)
0620 20:15:59.581936 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.581946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.581959 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:15:59.581995 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.582000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.582038 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.582606 (+ 568us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.582681 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.584108 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.585284 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.585666 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.586084 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.587861 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.587864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.587867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.587888 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.589883 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.590601 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.590604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.590606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.590685 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.590986 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.590989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.590990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.591230 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.591324 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.592049 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.592164 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063,"thread_start_us":105,"threads_started":1}
I20260620 20:15:59.594022 5755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:59.594894 5755 log.cc:824] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:59.595119 5679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066: Bootstrap starting.
I20260620 20:15:59.597049 5755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39: No bootstrap required, opened a new log
I20260620 20:15:59.597118 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:15:59.597476 5604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } attempt: 3
I20260620 20:15:59.597579 5679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:59.598641 5679 log.cc:824] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066: Log is configured to *not* fsync() on all Append() calls
I20260620 20:15:59.601811 5755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } }
I20260620 20:15:59.602079 5755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:59.602144 5755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a35728eff2a4b7580c39f7673d80f39, State: Initialized, Role: FOLLOWER
I20260620 20:15:59.602640 5755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } }
Jun 20 20:15:59 dist-test-slave-pflr krb5kdc[5522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986559, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.188@KRBTEST.COM
I20260620 20:15:59.603861 5679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066: No bootstrap required, opened a new log
I20260620 20:15:59.603896 5755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:59.607542 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.597874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:36349 (local address 127.0.0.1:59884)
0620 20:15:59.598020 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 20:15:59.598035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:15:59.598127 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:15:59.598594 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:15:59.598601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:15:59.598816 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:15:59.598941 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.598950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.599854 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.599859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.600431 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:15:59.600439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.601176 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.601179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:15:59.601298 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.601858 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:15:59.601876 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:15:59.603549 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:15:59.605849 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.605852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.605855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.606134 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.606445 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:15:59.606448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:15:59.606451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:15:59.606540 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:15:59.607011 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:15:59.607014 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:15:59.607112 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 20:15:59.607217 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 20:15:59.607364 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 20:15:59.608202 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.598159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.24.20.188:36349)
0620 20:15:59.598260 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.598265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.598277 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:15:59.598315 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.598318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.598370 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.598460 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.599090 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.599731 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.600558 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.600931 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.601042 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.603719 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.603723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.603725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.603750 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.605730 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.606254 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.606258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.606260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.606341 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.606640 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.606644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.606646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.606866 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.606966 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.607866 (+ 900us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.608004 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:15:59.608032 5679 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } }
I20260620 20:15:59.608274 5679 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:59.608328 5679 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e62fcd6bf74404ae9f70710cac0066, State: Initialized, Role: FOLLOWER
I20260620 20:15:59.608773 5679 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } }
I20260620 20:15:59.609982 5679 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:15:59.607450 5686 master_runner.cc:186] Error getting master registration for 127.24.20.190:33311: OK, instance_id { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" instance_seqno: 1781986559134009 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:15:59.611284 5686 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:15:59.611565 5604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5: Bootstrap starting.
I20260620 20:15:59.614075 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.601342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.24.20.188:36349)
0620 20:15:59.601809 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.601815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.602112 (+ 297us) server_negotiation.cc:407] Connection header received
0620 20:15:59.602708 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.602711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.602753 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.602808 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.604403 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.605609 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.606875 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.607223 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.607292 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.609937 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.609941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.609944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.609969 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.611857 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.612480 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.612484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.612486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.612571 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.612963 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.612967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.612969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.613181 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.613285 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.613821 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.613931 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
I20260620 20:15:59.614499 5604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5: Neither blocks nor log segments found. Creating new log.
I20260620 20:15:59.615442 5604 log.cc:824] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5: Log is configured to *not* fsync() on all Append() calls
W20260620 20:15:59.616295 5793 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:15:59.616364 5793 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:15:59.616524 5782 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } } }
I20260620 20:15:59.616657 5782 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:59.613407 5611 master_runner.cc:186] Error getting master registration for 127.24.20.190:33311: OK, instance_id { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" instance_seqno: 1781986559134009 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:15:59.617121 5611 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 20:15:59.618804 5781 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:15:59.618875 5781 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:15:59.624644 5604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5: No bootstrap required, opened a new log
I20260620 20:15:59.627033 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.601674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.24.20.190:33311)
0620 20:15:59.601775 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.601778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.614933 (+ 13155us) server_negotiation.cc:407] Connection header received
0620 20:15:59.616304 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.616308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.616358 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.616431 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.617695 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.618514 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.620027 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.620426 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.620520 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.622949 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.622954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.622961 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.622993 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.624954 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.625543 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.625547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.625549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.625636 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.625962 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.625966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.625968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.626177 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.626297 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.626750 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.626871 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:15:59.629402 5770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } } }
I20260620 20:15:59.629628 5770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:59.630656 5604 raft_consensus.cc:348] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } }
I20260620 20:15:59.630839 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:15:59.601004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.24.20.189:33005)
0620 20:15:59.601116 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 20:15:59.601123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:15:59.601776 (+ 653us) server_negotiation.cc:407] Connection header received
0620 20:15:59.604213 (+ 2437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:15:59.604217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:15:59.604277 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:15:59.604335 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:15:59.616637 (+ 12302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.617491 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.618778 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:15:59.619146 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:15:59.619227 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:15:59.621318 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:15:59.621322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:15:59.621325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:15:59.621353 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:15:59.623682 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.629114 (+ 5432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.629120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.629121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.629218 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:15:59.629572 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:15:59.629576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:15:59.629578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:15:59.629820 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:15:59.629999 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 20:15:59.630565 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 20:15:59.630691 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:15:59.630870 5604 raft_consensus.cc:374] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:15:59.630944 5604 raft_consensus.cc:729] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60f83cc9f44428ea3d458f3ce98bcd5, State: Initialized, Role: FOLLOWER
I20260620 20:15:59.632004 5604 consensus_queue.cc:260] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } }
I20260620 20:15:59.632520 5797 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } } }
I20260620 20:15:59.632741 5797 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:59.633419 5604 sys_catalog.cc:564] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:15:59.640311 5808 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:15:59.640389 5808 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:15:59.643342 5535 master_runner.cc:263] No leader master found from master d60f83cc9f44428ea3d458f3ce98bcd5
I20260620 20:15:59.643429 5535 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 20:15:59.788748 5707 tablet.cc:2369] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:15:59.828510 5632 tablet.cc:2369] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:15:59.830611 5797 raft_consensus.cc:482] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:15:59.830789 5797 raft_consensus.cc:504] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } }
I20260620 20:15:59.831741 5797 leader_election.cc:288] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4e62fcd6bf74404ae9f70710cac0066 (127.24.20.189:33005), 3a35728eff2a4b7580c39f7673d80f39 (127.24.20.188:36349)
I20260620 20:15:59.832300 5653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e62fcd6bf74404ae9f70710cac0066" is_pre_election: true
I20260620 20:15:59.832352 5729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a35728eff2a4b7580c39f7673d80f39" is_pre_election: true
I20260620 20:15:59.832644 5729 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d60f83cc9f44428ea3d458f3ce98bcd5 in term 0.
I20260620 20:15:59.832644 5653 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d60f83cc9f44428ea3d458f3ce98bcd5 in term 0.
I20260620 20:15:59.833081 5553 leader_election.cc:302] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a35728eff2a4b7580c39f7673d80f39, d60f83cc9f44428ea3d458f3ce98bcd5; no voters:
I20260620 20:15:59.833374 5797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:15:59.833463 5797 raft_consensus.cc:482] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:15:59.833513 5797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:59.834661 5797 raft_consensus.cc:504] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } }
I20260620 20:15:59.835033 5797 leader_election.cc:288] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [CANDIDATE]: Term 1 election: Requested vote from peers a4e62fcd6bf74404ae9f70710cac0066 (127.24.20.189:33005), 3a35728eff2a4b7580c39f7673d80f39 (127.24.20.188:36349)
I20260620 20:15:59.835394 5653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e62fcd6bf74404ae9f70710cac0066"
I20260620 20:15:59.835440 5729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a35728eff2a4b7580c39f7673d80f39"
I20260620 20:15:59.835546 5653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:59.835583 5729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:15:59.836539 5653 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d60f83cc9f44428ea3d458f3ce98bcd5 in term 1.
I20260620 20:15:59.836841 5553 leader_election.cc:302] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4e62fcd6bf74404ae9f70710cac0066, d60f83cc9f44428ea3d458f3ce98bcd5; no voters:
I20260620 20:15:59.836974 5729 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d60f83cc9f44428ea3d458f3ce98bcd5 in term 1.
I20260620 20:15:59.837020 5797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:15:59.837268 5797 raft_consensus.cc:686] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [term 1 LEADER]: Becoming Leader. State: Replica: d60f83cc9f44428ea3d458f3ce98bcd5, State: Running, Role: LEADER
I20260620 20:15:59.837610 5797 consensus_queue.cc:237] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } }
I20260620 20:15:59.839092 5810 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d60f83cc9f44428ea3d458f3ce98bcd5. Latest consensus state: current_term: 1 leader_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } } }
I20260620 20:15:59.839196 5810 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:59.839753 5812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:15:59.840557 5812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:15:59.844807 5729 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [term 1 FOLLOWER]: Refusing update from remote peer d60f83cc9f44428ea3d458f3ce98bcd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:59.844851 5653 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [term 1 FOLLOWER]: Refusing update from remote peer d60f83cc9f44428ea3d458f3ce98bcd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:15:59.845450 5797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:59.845597 5810 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:15:59.849182 5782 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d60f83cc9f44428ea3d458f3ce98bcd5. Latest consensus state: current_term: 1 leader_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } } }
I20260620 20:15:59.849201 5770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d60f83cc9f44428ea3d458f3ce98bcd5. Latest consensus state: current_term: 1 leader_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } } }
I20260620 20:15:59.849308 5770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:59.849308 5782 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:59.851063 5797 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } } }
I20260620 20:15:59.851172 5797 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:59.851485 5797 sys_catalog.cc:455] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } } }
I20260620 20:15:59.851586 5797 sys_catalog.cc:458] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:15:59.851576 5782 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } } }
I20260620 20:15:59.851680 5782 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:59.852520 5770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d60f83cc9f44428ea3d458f3ce98bcd5" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33311 } } peers { permanent_uuid: "a4e62fcd6bf74404ae9f70710cac0066" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 33005 } } peers { permanent_uuid: "3a35728eff2a4b7580c39f7673d80f39" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 36349 } } }
I20260620 20:15:59.852654 5770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:15:59.853454 5812 catalog_manager.cc:1347] Generated new cluster ID: 7dda22df2c0e48e0836dc89500c32c75
I20260620 20:15:59.853515 5812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:15:59.868050 5812 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:15:59.869138 5812 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:15:59.876057 5812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d60f83cc9f44428ea3d458f3ce98bcd5: Generated new TSK 0
I20260620 20:15:59.876984 5812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:16:00.617690 5793 catalog_manager.cc:1259] Loaded cluster ID: 7dda22df2c0e48e0836dc89500c32c75
I20260620 20:16:00.617776 5793 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066: loading cluster ID for follower catalog manager: success
I20260620 20:16:00.619872 5793 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066: acquiring CA information for follower catalog manager: success
I20260620 20:16:00.620169 5781 catalog_manager.cc:1259] Loaded cluster ID: 7dda22df2c0e48e0836dc89500c32c75
I20260620 20:16:00.620235 5781 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39: loading cluster ID for follower catalog manager: success
I20260620 20:16:00.621387 5793 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a4e62fcd6bf74404ae9f70710cac0066: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:16:00.622054 5781 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39: acquiring CA information for follower catalog manager: success
I20260620 20:16:00.623095 5781 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a35728eff2a4b7580c39f7673d80f39: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:16:00.931780 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 5535
I20260620 20:16:00.937904 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 5611
I20260620 20:16:00.943565 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 5686
2026-06-20T20:16:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3143 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:16:00 dist-test-slave-pflr krb5kdc[5827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:16:00 dist-test-slave-pflr krb5kdc[5827](info): set up 2 sockets
Jun 20 20:16:00 dist-test-slave-pflr krb5kdc[5827](info): commencing operation
krb5kdc: starting...
W20260620 20:16:01.985265 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:16:02 dist-test-slave-pflr krb5kdc[5827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:16:02.101423 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:35657
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:36741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:35657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:02.210855 5840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:02.211093 5840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:02.211133 5840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:02.215310 5840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:02.215368 5840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:02.215399 5840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:02.215415 5840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:02.215431 5840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:02.220701 5840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:35657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:35657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:02.221681 5840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:02.222668 5840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:02.230654 5845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:02.230669 5846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:02.230769 5848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:02.231072 5840 server_base.cc:1034] running on GCE node
I20260620 20:16:02.231582 5840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:02.232656 5840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:02.233834 5840 hybrid_clock.cc:648] HybridClock initialized: now 1781986562233827 us; error 30 us; skew 500 ppm
Jun 20 20:16:02 dist-test-slave-pflr krb5kdc[5827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986562, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:02.236500 5840 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:16:02.237326 5840 webserver.cc:460] Webserver started at http://127.24.20.190:39487/ using document root <none> and password file <none>
I20260620 20:16:02.237864 5840 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:02.237951 5840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:02.238173 5840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:02.240113 5840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "baa2739889084f629a0347daf89469c6"
format_stamp: "Formatted at 2026-06-20 20:16:02 on dist-test-slave-pflr"
server_key: "ffafc4d0d3fbe6685aa1cb96dbe08622"
server_key_iv: "c7762f7e70730c7ae48863f3befd8e1c"
server_key_version: "encryptionkey@0"
I20260620 20:16:02.240713 5840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "baa2739889084f629a0347daf89469c6"
format_stamp: "Formatted at 2026-06-20 20:16:02 on dist-test-slave-pflr"
server_key: "ffafc4d0d3fbe6685aa1cb96dbe08622"
server_key_iv: "c7762f7e70730c7ae48863f3befd8e1c"
server_key_version: "encryptionkey@0"
I20260620 20:16:02.244211 5840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:02.246500 5855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:02.247666 5840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:02.247893 5840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "baa2739889084f629a0347daf89469c6"
format_stamp: "Formatted at 2026-06-20 20:16:02 on dist-test-slave-pflr"
server_key: "ffafc4d0d3fbe6685aa1cb96dbe08622"
server_key_iv: "c7762f7e70730c7ae48863f3befd8e1c"
server_key_version: "encryptionkey@0"
I20260620 20:16:02.248003 5840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:02.264189 5840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:02.267370 5840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:02.267566 5840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:02.274379 5840 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:35657
I20260620 20:16:02.274392 5907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:35657 every 8 connection(s)
I20260620 20:16:02.275264 5840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:02.276755 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 5840
I20260620 20:16:02.276875 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:16:02.277173 24658 external_mini_cluster.cc:1428] Setting key d585eefaf9d1cc42708be1bcf1caac08
I20260620 20:16:02.278369 5908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:02.284988 5908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6: Bootstrap starting.
Jun 20 20:16:02 dist-test-slave-pflr krb5kdc[5827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:02.287519 5908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:02.288339 5908 log.cc:824] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:02.290464 5908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6: No bootstrap required, opened a new log
I20260620 20:16:02.291721 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:02.278485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.24.20.190:35657)
0620 20:16:02.278893 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:02.278916 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:02.278958 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:16:02.279807 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:02.279831 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:02.280237 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:02.280616 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:02.281292 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:02.282774 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:02.283615 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:02.284336 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:02.284490 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:02.287142 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:02.287147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:02.287159 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:02.287183 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:02.289099 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:02.289554 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:02.289558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:02.289561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:02.289660 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:02.289945 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:02.289947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:02.289948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:02.290301 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:02.290436 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:02.290767 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 20:16:02.291055 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":112,"threads_started":1}
I20260620 20:16:02.294402 5908 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa2739889084f629a0347daf89469c6" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 35657 } }
I20260620 20:16:02.294652 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:16:02.294689 5908 raft_consensus.cc:374] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:02.294736 5908 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa2739889084f629a0347daf89469c6, State: Initialized, Role: FOLLOWER
I20260620 20:16:02.295174 5908 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa2739889084f629a0347daf89469c6" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 35657 } }
I20260620 20:16:02.295339 5908 raft_consensus.cc:388] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:02.295405 5908 raft_consensus.cc:482] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:02.295504 5908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:02.296674 5908 raft_consensus.cc:504] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa2739889084f629a0347daf89469c6" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 35657 } }
I20260620 20:16:02.297029 5908 leader_election.cc:302] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baa2739889084f629a0347daf89469c6; no voters:
I20260620 20:16:02.297351 5908 leader_election.cc:288] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:02.297508 5913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:02.297767 5913 raft_consensus.cc:686] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [term 1 LEADER]: Becoming Leader. State: Replica: baa2739889084f629a0347daf89469c6, State: Running, Role: LEADER
I20260620 20:16:02.298110 5913 consensus_queue.cc:237] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa2739889084f629a0347daf89469c6" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 35657 } }
I20260620 20:16:02.298578 5908 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:02.299823 5915 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baa2739889084f629a0347daf89469c6. Latest consensus state: current_term: 1 leader_uuid: "baa2739889084f629a0347daf89469c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa2739889084f629a0347daf89469c6" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 35657 } } }
I20260620 20:16:02.299822 5914 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baa2739889084f629a0347daf89469c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa2739889084f629a0347daf89469c6" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 35657 } } }
I20260620 20:16:02.300092 5914 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:02.300235 5915 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:02.300551 5924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:02.303136 5924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:02.308493 5924 catalog_manager.cc:1347] Generated new cluster ID: d1b3973c403b4c28b1831f44abb4ba84
I20260620 20:16:02.308555 5924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:02.318428 5924 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:02.319590 5924 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:02.326558 5924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baa2739889084f629a0347daf89469c6: Generated new TSK 0
I20260620 20:16:02.327297 5924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:16:02.350662 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 5840
2026-06-20T20:16:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1405 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:16:02 dist-test-slave-pflr krb5kdc[5935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:16:02 dist-test-slave-pflr krb5kdc[5935](info): set up 2 sockets
Jun 20 20:16:02 dist-test-slave-pflr krb5kdc[5935](info): commencing operation
krb5kdc: starting...
W20260620 20:16:03.397094 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:16:03 dist-test-slave-pflr krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:16:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:16:03.517829 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:40511
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:40511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:03.631026 5948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:03.631304 5948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:03.631343 5948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:03.635267 5948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:03.635320 5948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:03.635340 5948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:03.635356 5948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:03.635371 5948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:03.640602 5948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:40511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:03.641613 5948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:03.642632 5948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:03.651420 5954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:03.651433 5953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:03.651363 5956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:03.651690 5948 server_base.cc:1034] running on GCE node
I20260620 20:16:03.652223 5948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:03.653491 5948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:03.654659 5948 hybrid_clock.cc:648] HybridClock initialized: now 1781986563654627 us; error 45 us; skew 500 ppm
Jun 20 20:16:03 dist-test-slave-pflr krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986563, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:03.657588 5948 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:16:03.658530 5948 webserver.cc:460] Webserver started at http://127.24.20.190:43539/ using document root <none> and password file <none>
I20260620 20:16:03.659116 5948 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:03.659209 5948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:03.659438 5948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:03.661568 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "fd14c2464f7b40abbdeb3c9db5575371"
format_stamp: "Formatted at 2026-06-20 20:16:03 on dist-test-slave-pflr"
server_key: "8593b1ba10690651e7f1cd84602ac9ca"
server_key_iv: "bcdc59d1db7e4fddc50b63e25685e9ca"
server_key_version: "encryptionkey@0"
I20260620 20:16:03.662144 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "fd14c2464f7b40abbdeb3c9db5575371"
format_stamp: "Formatted at 2026-06-20 20:16:03 on dist-test-slave-pflr"
server_key: "8593b1ba10690651e7f1cd84602ac9ca"
server_key_iv: "bcdc59d1db7e4fddc50b63e25685e9ca"
server_key_version: "encryptionkey@0"
I20260620 20:16:03.666702 5948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:16:03.669418 5963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:03.670732 5948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:16:03.670873 5948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "fd14c2464f7b40abbdeb3c9db5575371"
format_stamp: "Formatted at 2026-06-20 20:16:03 on dist-test-slave-pflr"
server_key: "8593b1ba10690651e7f1cd84602ac9ca"
server_key_iv: "bcdc59d1db7e4fddc50b63e25685e9ca"
server_key_version: "encryptionkey@0"
I20260620 20:16:03.671041 5948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:03.684404 5948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:03.687958 5948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:03.688189 5948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:03.695451 5948 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:40511
I20260620 20:16:03.695745 6015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:40511 every 8 connection(s)
I20260620 20:16:03.696508 5948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:03.699450 6016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:03.704089 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 5948
I20260620 20:16:03.704196 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:16:03.704480 24658 external_mini_cluster.cc:1428] Setting key afb99b903a432c7bcddbe7ae4a00e3e0
I20260620 20:16:03.705358 6016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Bootstrap starting.
I20260620 20:16:03.707638 6016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:03.708575 6016 log.cc:824] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:03.710726 6016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: No bootstrap required, opened a new log
Jun 20 20:16:03 dist-test-slave-pflr krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:03.715109 6016 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } }
I20260620 20:16:03.715363 6016 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:03.715423 6016 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd14c2464f7b40abbdeb3c9db5575371, State: Initialized, Role: FOLLOWER
I20260620 20:16:03.715868 6016 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } }
I20260620 20:16:03.715988 6016 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:03.716117 6016 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:03.716233 6016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:03.717167 6016 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } }
I20260620 20:16:03.717514 6016 leader_election.cc:302] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd14c2464f7b40abbdeb3c9db5575371; no voters:
I20260620 20:16:03.717854 6016 leader_election.cc:288] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:03.717939 6021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:03.718106 6021 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 1 LEADER]: Becoming Leader. State: Replica: fd14c2464f7b40abbdeb3c9db5575371, State: Running, Role: LEADER
I20260620 20:16:03.718392 6021 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } }
I20260620 20:16:03.719133 6016 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:03.719587 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd14c2464f7b40abbdeb3c9db5575371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } } }
I20260620 20:16:03.719700 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:03.719872 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:03.705985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.24.20.190:40511)
0620 20:16:03.706385 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:03.706401 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:03.706437 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:16:03.707088 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:03.707104 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:03.707431 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:03.707734 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:03.708699 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:03.709900 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:03.711069 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:03.711558 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:03.711677 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:03.714470 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:03.714475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:03.714487 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:03.714548 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:03.716539 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:03.717061 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:03.717066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:03.717074 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:03.717188 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:03.717492 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:03.717496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:03.717498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:03.718001 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:03.718200 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:03.718535 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 20:16:03.718846 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":143,"threads_started":1}
I20260620 20:16:03.719901 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd14c2464f7b40abbdeb3c9db5575371. Latest consensus state: current_term: 1 leader_uuid: "fd14c2464f7b40abbdeb3c9db5575371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } } }
I20260620 20:16:03.719990 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:03.720258 6025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:03.723882 6025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:03.729384 6025 catalog_manager.cc:1347] Generated new cluster ID: fdf419a5dc4148c9a3f00277e8aa69ec
I20260620 20:16:03.729460 6025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:03.736289 6025 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:03.737191 6025 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:03.748194 6025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Generated new TSK 0
I20260620 20:16:03.749043 6025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:16:03.784168 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:40511
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:03.897969 6042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:03.898211 6042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:03.898257 6042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:03.902493 6042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:03.902550 6042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:03.902606 6042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:16:03.907598 6042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:03.908573 6042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:03.909580 6042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:03.918982 6047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:03.919092 6042 server_base.cc:1034] running on GCE node
W20260620 20:16:03.918972 6048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:03.918970 6050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:03.919798 6042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:03.920423 6042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:03.921614 6042 hybrid_clock.cc:648] HybridClock initialized: now 1781986563921599 us; error 27 us; skew 500 ppm
Jun 20 20:16:03 dist-test-slave-pflr krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986563, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:03.924268 6042 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:16:03.925186 6042 webserver.cc:460] Webserver started at http://127.24.20.129:36275/ using document root <none> and password file <none>
I20260620 20:16:03.925732 6042 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:03.925820 6042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:03.926040 6042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:03.928010 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "f0c684e692bb4315a7b0134b14028767"
format_stamp: "Formatted at 2026-06-20 20:16:03 on dist-test-slave-pflr"
server_key: "539cfb23ceb1fe980e94516f94c26474"
server_key_iv: "13177ea81855668cd7aaeebb156ebf76"
server_key_version: "encryptionkey@0"
I20260620 20:16:03.928555 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "f0c684e692bb4315a7b0134b14028767"
format_stamp: "Formatted at 2026-06-20 20:16:03 on dist-test-slave-pflr"
server_key: "539cfb23ceb1fe980e94516f94c26474"
server_key_iv: "13177ea81855668cd7aaeebb156ebf76"
server_key_version: "encryptionkey@0"
I20260620 20:16:03.932080 6042 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:16:03.934437 6057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:03.935616 6042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:03.935740 6042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "f0c684e692bb4315a7b0134b14028767"
format_stamp: "Formatted at 2026-06-20 20:16:03 on dist-test-slave-pflr"
server_key: "539cfb23ceb1fe980e94516f94c26474"
server_key_iv: "13177ea81855668cd7aaeebb156ebf76"
server_key_version: "encryptionkey@0"
I20260620 20:16:03.935842 6042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:03.957417 6042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:03.960292 6042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:03.960497 6042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:03.961097 6042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:03.961951 6042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:03.962016 6042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:03.962080 6042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:03.962126 6042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:03.970989 6042 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:38101
I20260620 20:16:03.971017 6170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:38101 every 8 connection(s)
I20260620 20:16:03.971841 6042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:03.980422 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 6042
I20260620 20:16:03.980574 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:16:03.980921 24658 external_mini_cluster.cc:1428] Setting key 79b6d109e49bd4b224be7b45bee84e5e
Jun 20 20:16:03 dist-test-slave-pflr krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986563, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:03.985937 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:03.973544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:48641 (local address 127.24.20.190:40511)
0620 20:16:03.973691 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:03.973695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:03.974583 (+ 888us) server_negotiation.cc:407] Connection header received
0620 20:16:03.976119 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:03.976122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:03.976167 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:03.976242 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:03.977403 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:03.977928 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:03.978976 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:03.979194 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:03.979270 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:03.981944 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:03.981949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:03.981950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:03.981973 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:03.983655 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:03.984209 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:03.984216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:03.984220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:03.984303 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:03.984782 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:03.984785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:03.984786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:03.984964 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:03.985063 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:03.985717 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 20:16:03.985820 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:16:03.987099 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:03.974002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:48641)
0620 20:16:03.974417 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:03.974458 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:03.975936 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:03.976393 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:03.976400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:03.976804 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:03.977281 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:03.977293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:03.978027 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:03.978030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:03.978875 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:03.978883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:03.979314 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:03.979317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:03.979448 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:03.980029 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:03.980045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:03.981789 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:03.983766 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:03.983770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:03.983781 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:03.984080 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:03.984393 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:03.984398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:03.984401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:03.984654 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:03.985097 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:03.985105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:03.985419 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 20:16:03.985628 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 20:16:03.985975 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":135,"threads_started":1}
I20260620 20:16:03.988425 6171 heartbeater.cc:347] Connected to a master server at 127.24.20.190:40511
I20260620 20:16:03.988724 6171 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:03.989313 6171 heartbeater.cc:511] Master 127.24.20.190:40511 requested a full tablet report, sending...
I20260620 20:16:03.991078 5980 ts_manager.cc:194] Registered new tserver with Master: f0c684e692bb4315a7b0134b14028767 (127.24.20.129:38101)
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:16:03.992362 5980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:48641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:16:04.011441 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:40511
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:04.121145 6176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:04.121376 6176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:04.121416 6176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:04.125312 6176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:04.125379 6176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:04.125435 6176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:16:04.130483 6176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:04.131392 6176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:04.132318 6176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:04.141826 6182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:04.141844 6184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:04.141908 6176 server_base.cc:1034] running on GCE node
W20260620 20:16:04.141834 6181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:04.142395 6176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:04.143011 6176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:04.144196 6176 hybrid_clock.cc:648] HybridClock initialized: now 1781986564144188 us; error 27 us; skew 500 ppm
Jun 20 20:16:04 dist-test-slave-pflr krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986564, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:04.147023 6176 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:16:04.147868 6176 webserver.cc:460] Webserver started at http://127.24.20.130:43273/ using document root <none> and password file <none>
I20260620 20:16:04.148398 6176 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:04.148486 6176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:04.148772 6176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:04.150723 6176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-pflr"
server_key: "0502500b98e0a82f2865658b7ea163c9"
server_key_iv: "2cb510680ffbcc1ad3c49043a34a5adc"
server_key_version: "encryptionkey@0"
I20260620 20:16:04.151278 6176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-pflr"
server_key: "0502500b98e0a82f2865658b7ea163c9"
server_key_iv: "2cb510680ffbcc1ad3c49043a34a5adc"
server_key_version: "encryptionkey@0"
I20260620 20:16:04.154843 6176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:16:04.157250 6191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:04.158522 6176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:16:04.158619 6176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-pflr"
server_key: "0502500b98e0a82f2865658b7ea163c9"
server_key_iv: "2cb510680ffbcc1ad3c49043a34a5adc"
server_key_version: "encryptionkey@0"
I20260620 20:16:04.158746 6176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:04.188174 6176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:04.191795 6176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:04.192008 6176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:04.192679 6176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:04.193509 6176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:04.193570 6176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:04.193653 6176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:04.193691 6176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:04.202780 6176 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:45729
I20260620 20:16:04.202802 6304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:45729 every 8 connection(s)
I20260620 20:16:04.203665 6176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:04.212525 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 6176
I20260620 20:16:04.212685 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:16:04.212970 24658 external_mini_cluster.cc:1428] Setting key 2f287a21b2ca8205024f4fa1548b49e3
Jun 20 20:16:04 dist-test-slave-pflr krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986564, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:04.217552 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.205423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:49131 (local address 127.24.20.190:40511)
0620 20:16:04.205566 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:04.205570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:04.206361 (+ 791us) server_negotiation.cc:407] Connection header received
0620 20:16:04.207760 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:04.207763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:04.207818 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:04.207927 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:04.209108 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.209658 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.210791 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.211012 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.211131 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.213724 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:04.213729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:04.213732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:04.213759 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:04.215399 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:04.216027 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:04.216032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:04.216035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:04.216144 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:04.216458 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:04.216463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:04.216466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:04.216742 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:04.216855 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:04.217338 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 20:16:04.217435 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:16:04.218768 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.205730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:49131)
0620 20:16:04.206196 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:04.206237 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:04.207548 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:04.208122 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:04.208129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:04.208474 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:04.208963 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.208975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.209798 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.209801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.210656 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.210665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.211179 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.211183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.211341 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.211854 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:04.211868 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:04.213505 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:04.215514 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:04.215522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:04.215547 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:04.215910 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:04.216240 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:04.216243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:04.216244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:04.216367 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:04.216846 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:04.216851 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:04.217065 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 20:16:04.217272 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 20:16:04.217614 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":104,"threads_started":1}
I20260620 20:16:04.220059 6305 heartbeater.cc:347] Connected to a master server at 127.24.20.190:40511
I20260620 20:16:04.220386 6305 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:04.220988 6305 heartbeater.cc:511] Master 127.24.20.190:40511 requested a full tablet report, sending...
I20260620 20:16:04.222146 5980 ts_manager.cc:194] Registered new tserver with Master: df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729)
I20260620 20:16:04.222736 5980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:49131
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:16:04.239578 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:40511
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:04.347791 6310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:04.348066 6310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:04.348184 6310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:04.352104 6310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:04.352185 6310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:04.352278 6310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:16:04.357460 6310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:04.358407 6310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:04.359350 6310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:04.368470 6318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:04.368470 6316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:04.368534 6315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:04.368722 6310 server_base.cc:1034] running on GCE node
I20260620 20:16:04.369202 6310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:04.369843 6310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:04.371033 6310 hybrid_clock.cc:648] HybridClock initialized: now 1781986564371011 us; error 40 us; skew 500 ppm
Jun 20 20:16:04 dist-test-slave-pflr krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986564, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:04.373725 6310 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:16:04.374679 6310 webserver.cc:460] Webserver started at http://127.24.20.131:40531/ using document root <none> and password file <none>
I20260620 20:16:04.375202 6310 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:04.375288 6310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:04.375486 6310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:04.377429 6310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "c2370e07aa23459da0ffbf0ae47d72d6"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-pflr"
server_key: "825463e52c44bce65ea7a84fbbfeadf5"
server_key_iv: "8bea81ecff9ca4b2f75a0c5c895c2052"
server_key_version: "encryptionkey@0"
I20260620 20:16:04.377969 6310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "c2370e07aa23459da0ffbf0ae47d72d6"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-pflr"
server_key: "825463e52c44bce65ea7a84fbbfeadf5"
server_key_iv: "8bea81ecff9ca4b2f75a0c5c895c2052"
server_key_version: "encryptionkey@0"
I20260620 20:16:04.381368 6310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:04.383518 6325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:04.384743 6310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:16:04.384855 6310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "c2370e07aa23459da0ffbf0ae47d72d6"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-pflr"
server_key: "825463e52c44bce65ea7a84fbbfeadf5"
server_key_iv: "8bea81ecff9ca4b2f75a0c5c895c2052"
server_key_version: "encryptionkey@0"
I20260620 20:16:04.384963 6310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:04.398396 6310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:04.401206 6310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:04.401386 6310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:04.401904 6310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:04.402675 6310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:04.402745 6310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:04.402809 6310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:04.402863 6310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:04.411864 6310 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:35197
I20260620 20:16:04.411890 6438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:35197 every 8 connection(s)
I20260620 20:16:04.412765 6310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:04.415131 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 6310
I20260620 20:16:04.415261 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:16:04.415560 24658 external_mini_cluster.cc:1428] Setting key a87e49cf066e96cc748d826591d487df
Jun 20 20:16:04 dist-test-slave-pflr krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986564, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:04.426173 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.414444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:46023 (local address 127.24.20.190:40511)
0620 20:16:04.414619 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:04.414622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:04.415415 (+ 793us) server_negotiation.cc:407] Connection header received
0620 20:16:04.417140 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:04.417143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:04.417184 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:04.417251 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:04.418468 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.419037 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.420282 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.420490 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.420577 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.423114 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:04.423117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:04.423119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:04.423140 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:04.424361 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:04.424832 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:04.424836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:04.424837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:04.424935 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:04.425242 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:04.425244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:04.425245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:04.425428 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:04.425518 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:04.425947 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 20:16:04.426049 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:16:04.426877 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.414806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:46023)
0620 20:16:04.415254 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:04.415304 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:04.416910 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:04.417418 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:04.417426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:04.417896 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:04.418345 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.418358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.419140 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.419143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.420180 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.420190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.420667 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.420674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.420843 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.421428 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:04.421443 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:04.422959 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:04.424461 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:04.424465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:04.424476 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:04.424736 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:04.425026 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:04.425028 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:04.425029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:04.425153 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:04.425519 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:04.425524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:04.425726 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 20:16:04.425864 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 20:16:04.426101 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":127,"threads_started":1}
I20260620 20:16:04.427989 6439 heartbeater.cc:347] Connected to a master server at 127.24.20.190:40511
I20260620 20:16:04.428223 6439 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:04.428732 6439 heartbeater.cc:511] Master 127.24.20.190:40511 requested a full tablet report, sending...
I20260620 20:16:04.429675 5980 ts_manager.cc:194] Registered new tserver with Master: c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197)
I20260620 20:16:04.430224 5980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:46023
I20260620 20:16:04.441051 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:16:04.450320 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.442893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.24.20.190:40511)
0620 20:16:04.443041 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:04.443046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:04.443221 (+ 175us) server_negotiation.cc:407] Connection header received
0620 20:16:04.443354 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:04.443357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:04.443395 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:04.443484 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:04.444181 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.444796 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.445629 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.445852 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.445957 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.447187 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:04.447191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:04.447194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:04.447217 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:04.448683 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:04.449087 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:04.449091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:04.449093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:04.449177 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:04.449455 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:04.449458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:04.449460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:04.449668 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:04.449805 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:04.450045 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 20:16:04.450160 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:16:04.452934 5980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:16:04.455840 5980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:16:04.470476 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.464133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:35197 (local address 127.0.0.1:51124)
0620 20:16:04.465398 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:04.465447 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:04.465610 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:04.466310 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:04.466313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:04.466337 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:04.466444 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.466450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.467599 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.467603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.468916 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.468926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.469904 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.469907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.470018 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.470087 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 20:16:04.470193 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:16:04.470307 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1085,"mutex_wait_us":86,"thread_start_us":61,"threads_started":1}
I20260620 20:16:04.470537 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.464255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.24.20.131:35197)
0620 20:16:04.465061 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:04.465073 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:04.465540 (+ 467us) server_negotiation.cc:407] Connection header received
0620 20:16:04.465771 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:04.465777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:04.465982 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:04.466152 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:04.466585 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.467413 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.469039 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.469774 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.469887 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.470029 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:04.470269 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 20:16:04.470384 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":163,"threads_started":1}
I20260620 20:16:04.470665 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.464218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:38101 (local address 127.0.0.1:56652)
0620 20:16:04.465478 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:04.465492 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:04.465610 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:04.466152 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:04.466157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:04.466180 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:04.466307 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.466316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.467678 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.467681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.469214 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.469224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.470298 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.470301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.470377 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.470397 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:16:04.470461 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 20:16:04.470541 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1199,"mutex_wait_us":138,"spinlock_wait_cycles":40320,"thread_start_us":88,"threads_started":1}
I20260620 20:16:04.471020 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.464915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.24.20.129:38101)
0620 20:16:04.465239 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:04.465252 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:04.465598 (+ 346us) server_negotiation.cc:407] Connection header received
0620 20:16:04.465771 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:04.465777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:04.465982 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:04.466152 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:04.466434 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.467516 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.469350 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.470005 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.470105 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.470241 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:04.470829 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 20:16:04.470913 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":104,"threads_started":1}
I20260620 20:16:04.472936 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.464870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.24.20.130:45729)
0620 20:16:04.465267 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:04.465279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:04.465926 (+ 647us) server_negotiation.cc:407] Connection header received
0620 20:16:04.466098 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:04.466105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:04.466304 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:04.466460 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:04.466885 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.467984 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.469451 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.470284 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.470384 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.470503 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:04.472698 (+ 2195us) server_negotiation.cc:299] Negotiation successful
0620 20:16:04.472796 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":107,"threads_started":1}
I20260620 20:16:04.473174 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.464473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:45729 (local address 127.0.0.1:42186)
0620 20:16:04.465827 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:04.465836 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:04.465937 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:04.466662 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:04.466665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:04.466681 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:04.466757 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.466763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.468139 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.468142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.469339 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.469347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.471343 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.471347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.471447 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.471472 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:16:04.471548 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 20:16:04.471633 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1307,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260620 20:16:04.473623 6373 tablet_service.cc:1467] Processing CreateTablet for tablet a3d61a04afbe47de9516068f16ba4d7d (DEFAULT_TABLE table=test-table [id=6a4605007bf349a1b34f414e52d57c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:04.473624 6105 tablet_service.cc:1467] Processing CreateTablet for tablet a3d61a04afbe47de9516068f16ba4d7d (DEFAULT_TABLE table=test-table [id=6a4605007bf349a1b34f414e52d57c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:04.474587 6105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d61a04afbe47de9516068f16ba4d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:04.474588 6373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d61a04afbe47de9516068f16ba4d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:04.480489 6457 tablet_bootstrap.cc:492] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Bootstrap starting.
I20260620 20:16:04.481141 6456 tablet_bootstrap.cc:492] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6: Bootstrap starting.
I20260620 20:16:04.482847 6456 tablet_bootstrap.cc:654] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:04.482844 6457 tablet_bootstrap.cc:654] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:04.483330 6239 tablet_service.cc:1467] Processing CreateTablet for tablet a3d61a04afbe47de9516068f16ba4d7d (DEFAULT_TABLE table=test-table [id=6a4605007bf349a1b34f414e52d57c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:04.483984 6456 log.cc:824] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:04.484081 6457 log.cc:824] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:04.484421 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d61a04afbe47de9516068f16ba4d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:04.485843 6456 tablet_bootstrap.cc:492] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6: No bootstrap required, opened a new log
I20260620 20:16:04.485939 6457 tablet_bootstrap.cc:492] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: No bootstrap required, opened a new log
I20260620 20:16:04.486120 6457 ts_tablet_manager.cc:1397] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:16:04.486122 6456 ts_tablet_manager.cc:1397] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:16:04.489526 6456 raft_consensus.cc:348] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } }
I20260620 20:16:04.489754 6456 raft_consensus.cc:374] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:04.489806 6456 raft_consensus.cc:729] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2370e07aa23459da0ffbf0ae47d72d6, State: Initialized, Role: FOLLOWER
I20260620 20:16:04.489817 6457 raft_consensus.cc:348] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } }
I20260620 20:16:04.490067 6457 raft_consensus.cc:374] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:04.490124 6457 raft_consensus.cc:729] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c684e692bb4315a7b0134b14028767, State: Initialized, Role: FOLLOWER
I20260620 20:16:04.490259 6456 consensus_queue.cc:260] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } }
I20260620 20:16:04.490528 6457 consensus_queue.cc:260] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } }
I20260620 20:16:04.490934 6439 heartbeater.cc:503] Master 127.24.20.190:40511 was elected leader, sending a full tablet report...
I20260620 20:16:04.491250 6456 ts_tablet_manager.cc:1428] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:16:04.491376 6457 ts_tablet_manager.cc:1428] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:16:04.491681 6171 heartbeater.cc:503] Master 127.24.20.190:40511 was elected leader, sending a full tablet report...
I20260620 20:16:04.492125 6462 tablet_bootstrap.cc:492] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Bootstrap starting.
I20260620 20:16:04.494661 6462 tablet_bootstrap.cc:654] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:04.495466 6462 log.cc:824] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:04.497601 6462 tablet_bootstrap.cc:492] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: No bootstrap required, opened a new log
I20260620 20:16:04.497802 6462 ts_tablet_manager.cc:1397] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:16:04.500710 6462 raft_consensus.cc:348] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } }
I20260620 20:16:04.500929 6462 raft_consensus.cc:374] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:04.500988 6462 raft_consensus.cc:729] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3b718d1ec14a2ca2ee1fc0e47ba9b4, State: Initialized, Role: FOLLOWER
I20260620 20:16:04.501466 6462 consensus_queue.cc:260] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } }
I20260620 20:16:04.502117 6305 heartbeater.cc:503] Master 127.24.20.190:40511 was elected leader, sending a full tablet report...
I20260620 20:16:04.502403 6462 ts_tablet_manager.cc:1428] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:16:04.607228 6461 raft_consensus.cc:482] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:04.607476 6461 raft_consensus.cc:504] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } }
I20260620 20:16:04.608721 6461 leader_election.cc:288] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197), df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729)
I20260620 20:16:04.612915 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.609122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:39811 (local address 127.24.20.130:45729)
0620 20:16:04.609243 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:04.609248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:04.609629 (+ 381us) server_negotiation.cc:407] Connection header received
0620 20:16:04.609741 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:04.609743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:04.609793 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:04.609861 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:04.610133 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.610642 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.611862 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.612296 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.612393 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.612435 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:04.612716 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 20:16:04.612789 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:16:04.612949 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.608988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:45729 (local address 127.24.20.129:39811)
0620 20:16:04.609472 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:04.609489 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:04.609636 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:04.609888 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:04.609890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:04.609905 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:04.610010 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.610015 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.610761 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.610766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.611745 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.611753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.612423 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.612427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.612567 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.612630 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 20:16:04.612697 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 20:16:04.612766 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":19,"spinlock_wait_cycles":19072,"thread_start_us":186,"threads_started":1}
I20260620 20:16:04.613780 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.609100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:53767 (local address 127.24.20.131:35197)
0620 20:16:04.609239 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:04.609244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:04.609589 (+ 345us) server_negotiation.cc:407] Connection header received
0620 20:16:04.609980 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:04.609982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:04.610022 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:04.610091 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:04.611031 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.611500 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.612659 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.613200 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.613312 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.613362 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:04.613606 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 20:16:04.613676 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:16:04.613794 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.608998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:35197 (local address 127.24.20.129:53767)
0620 20:16:04.609472 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:04.609489 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:04.609636 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:04.610788 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:04.610790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:04.610810 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:04.610930 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.610936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.611609 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.611612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.612524 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:04.612534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.613352 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.613356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:04.613489 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.613511 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:16:04.613589 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 20:16:04.613654 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":25,"spinlock_wait_cycles":14336,"thread_start_us":221,"threads_started":1}
I20260620 20:16:04.613739 6259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d61a04afbe47de9516068f16ba4d7d" candidate_uuid: "f0c684e692bb4315a7b0134b14028767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" is_pre_election: true
I20260620 20:16:04.614076 6259 raft_consensus.cc:2454] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0c684e692bb4315a7b0134b14028767 in term 0.
I20260620 20:16:04.614351 6393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d61a04afbe47de9516068f16ba4d7d" candidate_uuid: "f0c684e692bb4315a7b0134b14028767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" is_pre_election: true
I20260620 20:16:04.614585 6060 leader_election.cc:302] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df3b718d1ec14a2ca2ee1fc0e47ba9b4, f0c684e692bb4315a7b0134b14028767; no voters:
I20260620 20:16:04.614675 6393 raft_consensus.cc:2454] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0c684e692bb4315a7b0134b14028767 in term 0.
I20260620 20:16:04.614864 6461 raft_consensus.cc:2781] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:16:04.614965 6461 raft_consensus.cc:482] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:16:04.615024 6461 raft_consensus.cc:3037] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:04.616042 6461 raft_consensus.cc:504] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } }
I20260620 20:16:04.616375 6461 leader_election.cc:288] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 1 election: Requested vote from peers c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197), df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729)
I20260620 20:16:04.616731 6393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d61a04afbe47de9516068f16ba4d7d" candidate_uuid: "f0c684e692bb4315a7b0134b14028767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2370e07aa23459da0ffbf0ae47d72d6"
I20260620 20:16:04.616768 6259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d61a04afbe47de9516068f16ba4d7d" candidate_uuid: "f0c684e692bb4315a7b0134b14028767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4"
I20260620 20:16:04.616825 6393 raft_consensus.cc:3037] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:04.616909 6259 raft_consensus.cc:3037] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:04.617810 6393 raft_consensus.cc:2454] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0c684e692bb4315a7b0134b14028767 in term 1.
I20260620 20:16:04.617957 6259 raft_consensus.cc:2454] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0c684e692bb4315a7b0134b14028767 in term 1.
I20260620 20:16:04.618103 6058 leader_election.cc:302] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2370e07aa23459da0ffbf0ae47d72d6, f0c684e692bb4315a7b0134b14028767; no voters:
I20260620 20:16:04.618263 6461 raft_consensus.cc:2781] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:04.618482 6461 raft_consensus.cc:686] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 1 LEADER]: Becoming Leader. State: Replica: f0c684e692bb4315a7b0134b14028767, State: Running, Role: LEADER
I20260620 20:16:04.618826 6461 consensus_queue.cc:237] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } }
I20260620 20:16:04.622186 5979 catalog_manager.cc:5495] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0c684e692bb4315a7b0134b14028767 (127.24.20.129). New cstate: current_term: 1 leader_uuid: "f0c684e692bb4315a7b0134b14028767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } health_report { overall_health: UNKNOWN } } }
I20260620 20:16:04.658524 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.655189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.24.20.129:38101)
0620 20:16:04.655310 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:04.655313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:04.655326 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:16:04.655395 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:04.655397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:04.655447 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:04.655541 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:04.655740 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.656223 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.657065 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:04.657273 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:04.657353 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:04.657705 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:04.658110 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:04.658199 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:04.658309 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 20:16:04.658354 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 20:16:04.663669 6440 tablet.cc:2369] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:16:04.664870 6393 raft_consensus.cc:1261] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6 [term 1 FOLLOWER]: Refusing update from remote peer f0c684e692bb4315a7b0134b14028767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:04.664870 6259 raft_consensus.cc:1261] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 1 FOLLOWER]: Refusing update from remote peer f0c684e692bb4315a7b0134b14028767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:04.665584 6467 consensus_queue.cc:1035] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:04.665773 6461 consensus_queue.cc:1035] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:04.673671 6472 mvcc.cc:205] Tried to move back new op lower bound from 7299016968859344896 to 7299016968681570304. Current Snapshot: MvccSnapshot[applied={T|T < 7299016968859344896}]
I20260620 20:16:04.675511 6471 mvcc.cc:205] Tried to move back new op lower bound from 7299016968859344896 to 7299016968681570304. Current Snapshot: MvccSnapshot[applied={T|T < 7299016968859344896}]
I20260620 20:16:04.680958 5979 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:04.681097 5979 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:04.683328 5979 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Sending DeleteTablet for 3 replicas of tablet a3d61a04afbe47de9516068f16ba4d7d
I20260620 20:16:04.683988 6373 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d61a04afbe47de9516068f16ba4d7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:04 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:51124
I20260620 20:16:04.684211 6239 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d61a04afbe47de9516068f16ba4d7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:04 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:42186
I20260620 20:16:04.684216 6105 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d61a04afbe47de9516068f16ba4d7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:04 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:56652
I20260620 20:16:04.684438 6479 tablet_replica.cc:330] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6: stopping tablet replica
I20260620 20:16:04.684744 6479 raft_consensus.cc:2229] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:04.684886 6481 tablet_replica.cc:330] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: stopping tablet replica
I20260620 20:16:04.684911 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 6042
I20260620 20:16:04.685918 6479 raft_consensus.cc:2258] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:04.687237 6479 ts_tablet_manager.cc:1905] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:04.689859 6479 ts_tablet_manager.cc:1918] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:04.689937 6479 log.cc:1196] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/a3d61a04afbe47de9516068f16ba4d7d
I20260620 20:16:04.690276 6479 ts_tablet_manager.cc:1939] T a3d61a04afbe47de9516068f16ba4d7d P c2370e07aa23459da0ffbf0ae47d72d6: Deleting consensus metadata
I20260620 20:16:04.690683 6480 tablet_replica.cc:330] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: stopping tablet replica
W20260620 20:16:04.690673 5967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:38101 (error 108)
I20260620 20:16:04.690956 6480 raft_consensus.cc:2229] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:04.691246 6480 raft_consensus.cc:2258] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:04.691802 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.691480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:38101 (local address 127.0.0.1:56676)
0620 20:16:04.691653 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:04.691717 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:38101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260620 20:16:04.691983 5967 catalog_manager.cc:4584] TS f0c684e692bb4315a7b0134b14028767 (127.24.20.129:38101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3d61a04afbe47de9516068f16ba4d7d: Network error: Client connection negotiation failed: client connection to 127.24.20.129:38101: connect: Connection refused (error 111)
I20260620 20:16:04.692029 5964 catalog_manager.cc:4841] TS c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197): tablet a3d61a04afbe47de9516068f16ba4d7d (table test-table [id=6a4605007bf349a1b34f414e52d57c32]) successfully deleted
I20260620 20:16:04.692108 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 6176
I20260620 20:16:04.697379 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 6310
I20260620 20:16:04.698181 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:04.697485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:45729 (local address 127.0.0.1:42188)
0620 20:16:04.698024 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:04.698095 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.130:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448}
I20260620 20:16:04.704268 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 5948
I20260620 20:16:04.710525 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:40511
--webserver_interface=127.24.20.190
--webserver_port=43539
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:40511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:04.817440 6483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:04.817737 6483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:04.817806 6483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:04.821802 6483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:04.821897 6483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:04.821935 6483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:04.821979 6483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:04.822013 6483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:04.827152 6483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:40511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=43539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:04.828233 6483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:04.829283 6483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:04.841804 6490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:04.841852 6489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:04.841931 6483 server_base.cc:1034] running on GCE node
W20260620 20:16:04.841806 6492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:04.842849 6483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:04.844003 6483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:04.845172 6483 hybrid_clock.cc:648] HybridClock initialized: now 1781986564845156 us; error 29 us; skew 500 ppm
Jun 20 20:16:04 dist-test-slave-pflr krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986564, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:04.848166 6483 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:16:04.849177 6483 webserver.cc:460] Webserver started at http://127.24.20.190:43539/ using document root <none> and password file <none>
I20260620 20:16:04.849743 6483 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:04.849834 6483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:04.853876 6483 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:04.856143 6499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:04.857648 6483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:16:04.857788 6483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "fd14c2464f7b40abbdeb3c9db5575371"
format_stamp: "Formatted at 2026-06-20 20:16:03 on dist-test-slave-pflr"
server_key: "8593b1ba10690651e7f1cd84602ac9ca"
server_key_iv: "bcdc59d1db7e4fddc50b63e25685e9ca"
server_key_version: "encryptionkey@0"
I20260620 20:16:04.858425 6483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:04.880514 6483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:04.883541 6483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:04.883738 6483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:04.891094 6483 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:40511
I20260620 20:16:04.891367 6551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:40511 every 8 connection(s)
I20260620 20:16:04.892270 6483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:04.896342 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 6483
I20260620 20:16:04.896637 6552 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:16:04.896870 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:38101
--local_ip_for_outbound_sockets=127.24.20.129
--tserver_master_addrs=127.24.20.190:40511
--webserver_port=36275
--webserver_interface=127.24.20.129
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:16:04.898379 6552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Bootstrap starting.
I20260620 20:16:04.915246 6552 log.cc:824] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:04.923810 6552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:16:04.924190 6552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Bootstrap complete.
I20260620 20:16:04.928195 6552 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } }
I20260620 20:16:04.928596 6552 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd14c2464f7b40abbdeb3c9db5575371, State: Initialized, Role: FOLLOWER
I20260620 20:16:04.929148 6552 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } }
I20260620 20:16:04.929292 6552 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:04.929385 6552 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:04.929504 6552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:16:04.931308 6552 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } }
I20260620 20:16:04.931640 6552 leader_election.cc:302] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd14c2464f7b40abbdeb3c9db5575371; no voters:
I20260620 20:16:04.931978 6552 leader_election.cc:288] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:16:04.932055 6556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:16:04.932243 6556 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 2 LEADER]: Becoming Leader. State: Replica: fd14c2464f7b40abbdeb3c9db5575371, State: Running, Role: LEADER
I20260620 20:16:04.932541 6556 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } }
I20260620 20:16:04.933204 6552 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:04.933450 6556 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd14c2464f7b40abbdeb3c9db5575371. Latest consensus state: current_term: 2 leader_uuid: "fd14c2464f7b40abbdeb3c9db5575371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } } }
I20260620 20:16:04.933565 6556 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:04.933962 6560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:04.933954 6557 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fd14c2464f7b40abbdeb3c9db5575371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } } }
I20260620 20:16:04.934182 6557 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:04.938694 6560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:04.939306 6560 catalog_manager.cc:1259] Loaded cluster ID: fdf419a5dc4148c9a3f00277e8aa69ec
I20260620 20:16:04.939368 6560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:04.940784 6560 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 20:16:04.945657 6560 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 20:16:04.950984 6560 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:04.951850 6560 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Loaded TSK: 0
I20260620 20:16:04.953102 6560 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:16:05.010630 6554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:05.010890 6554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:05.010965 6554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:05.015029 6554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:05.015110 6554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:05.015199 6554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:16:05.020310 6554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:38101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=36275
--tserver_master_addrs=127.24.20.190:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:05.021353 6554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:05.022430 6554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:05.031737 6579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:05.031694 6582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:05.031658 6580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:05.031939 6554 server_base.cc:1034] running on GCE node
I20260620 20:16:05.032516 6554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:05.033304 6554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:05.034497 6554 hybrid_clock.cc:648] HybridClock initialized: now 1781986565034455 us; error 47 us; skew 500 ppm
Jun 20 20:16:05 dist-test-slave-pflr krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986565, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:05.037773 6554 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:16:05.038806 6554 webserver.cc:460] Webserver started at http://127.24.20.129:36275/ using document root <none> and password file <none>
I20260620 20:16:05.039474 6554 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:05.039587 6554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:05.044517 6554 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:16:05.047130 6589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:05.048697 6554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:16:05.048818 6554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "f0c684e692bb4315a7b0134b14028767"
format_stamp: "Formatted at 2026-06-20 20:16:03 on dist-test-slave-pflr"
server_key: "539cfb23ceb1fe980e94516f94c26474"
server_key_iv: "13177ea81855668cd7aaeebb156ebf76"
server_key_version: "encryptionkey@0"
I20260620 20:16:05.049561 6554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:05.073259 6554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:05.076979 6554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:05.077181 6554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:05.077843 6554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:05.079198 6596 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 20:16:05.085168 6554 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 20:16:05.085322 6554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 20:16:05.085472 6554 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 20:16:05.087743 6554 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 20:16:05.087877 6554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:05.088093 6596 tablet_bootstrap.cc:492] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Bootstrap starting.
I20260620 20:16:05.097862 6554 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:38101
I20260620 20:16:05.099006 6554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:05.099591 6703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:38101 every 8 connection(s)
I20260620 20:16:05.103335 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 6554
I20260620 20:16:05.103775 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:45729
--local_ip_for_outbound_sockets=127.24.20.130
--tserver_master_addrs=127.24.20.190:40511
--webserver_port=43273
--webserver_interface=127.24.20.130
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:16:05.108696 6596 log.cc:824] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:16:05 dist-test-slave-pflr krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986565, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:05.117383 6596 tablet_bootstrap.cc:492] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:16:05.117754 6596 tablet_bootstrap.cc:492] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Bootstrap complete.
I20260620 20:16:05.119113 6596 ts_tablet_manager.cc:1397] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.014s
I20260620 20:16:05.119443 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.101320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:34891)
0620 20:16:05.102049 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:05.102102 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:05.102901 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:05.104387 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:05.104398 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:05.104955 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:05.105672 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.105696 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.107708 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.107712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.109061 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.109073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.109867 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.109874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.110124 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.110892 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:05.110919 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:05.113063 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:05.115770 (+ 2707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:05.115777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:05.115791 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:05.116136 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:05.116552 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:05.116555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:05.116557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:05.116748 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:05.117534 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:05.117540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:05.117902 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 20:16:05.118094 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 20:16:05.118429 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":147,"threads_started":1}
I20260620 20:16:05.119455 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.101381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:34891 (local address 127.24.20.190:40511)
0620 20:16:05.101766 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:05.101785 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:05.102244 (+ 459us) server_negotiation.cc:407] Connection header received
0620 20:16:05.103404 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:05.103423 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:05.103903 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:05.104297 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:05.105888 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.107524 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.109176 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.109676 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.109883 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.113305 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:05.113314 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:05.113329 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:05.113363 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:05.115635 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.116278 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.116283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.116289 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.116447 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.116862 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.116865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.116867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.117319 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:05.117455 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:05.118258 (+ 803us) server_negotiation.cc:299] Negotiation successful
0620 20:16:05.118603 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":122,"threads_started":1}
I20260620 20:16:05.121732 6596 raft_consensus.cc:348] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } }
I20260620 20:16:05.122251 6596 raft_consensus.cc:729] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c684e692bb4315a7b0134b14028767, State: Initialized, Role: FOLLOWER
I20260620 20:16:05.122414 6704 heartbeater.cc:347] Connected to a master server at 127.24.20.190:40511
I20260620 20:16:05.122623 6704 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:05.122731 6596 consensus_queue.cc:260] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } }
I20260620 20:16:05.123147 6704 heartbeater.cc:511] Master 127.24.20.190:40511 requested a full tablet report, sending...
I20260620 20:16:05.123692 6596 ts_tablet_manager.cc:1428] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 20:16:05.125422 6516 ts_manager.cc:194] Registered new tserver with Master: f0c684e692bb4315a7b0134b14028767 (127.24.20.129:38101)
I20260620 20:16:05.128681 6516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:34891
I20260620 20:16:05.130790 6704 heartbeater.cc:503] Master 127.24.20.190:40511 was elected leader, sending a full tablet report...
Jun 20 20:16:05 dist-test-slave-pflr krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986564, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.129@KRBTEST.COM
I20260620 20:16:05.138357 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.128004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:38101 (local address 127.0.0.1:56678)
0620 20:16:05.128532 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:05.128572 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:05.128712 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:05.129262 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:05.129267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:05.129646 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:05.129764 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.129772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.131105 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.131109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.131985 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.131995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.132702 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.132706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.132858 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.133393 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:05.133404 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:05.134743 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:05.136697 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:05.136706 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:05.136711 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:05.136932 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:05.137230 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:05.137234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:05.137236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:05.137321 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:05.137744 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:05.137749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:05.137870 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 20:16:05.138000 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 20:16:05.138185 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":110,"threads_started":1}
I20260620 20:16:05.138406 6712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.128095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.24.20.129:38101)
0620 20:16:05.128758 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:05.128833 (+ 75us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:05.128855 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:16:05.128917 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:05.128923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:05.129107 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:05.129235 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:05.129891 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.130985 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.132177 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.132544 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.132701 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.134878 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:05.134891 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:05.134899 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:05.134919 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:05.136544 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.137048 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.137054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.137056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.137135 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.137409 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.137414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.137416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.137632 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:05.137769 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:05.138108 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 20:16:05.138255 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":574,"thread_start_us":92,"threads_started":1}
I20260620 20:16:05.139415 6629 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d61a04afbe47de9516068f16ba4d7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:04 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:56678
I20260620 20:16:05.140208 6713 tablet_replica.cc:330] stopping tablet replica
I20260620 20:16:05.140513 6713 raft_consensus.cc:2229] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:05.140781 6713 raft_consensus.cc:2258] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:05.141806 6713 ts_tablet_manager.cc:1905] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:05.146100 6713 ts_tablet_manager.cc:1918] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:05.146198 6713 log.cc:1196] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal/wals/a3d61a04afbe47de9516068f16ba4d7d
I20260620 20:16:05.146674 6713 ts_tablet_manager.cc:1939] T a3d61a04afbe47de9516068f16ba4d7d P f0c684e692bb4315a7b0134b14028767: Deleting consensus metadata
I20260620 20:16:05.148746 6503 catalog_manager.cc:4841] TS f0c684e692bb4315a7b0134b14028767 (127.24.20.129:38101): tablet a3d61a04afbe47de9516068f16ba4d7d (table test-table [id=6a4605007bf349a1b34f414e52d57c32]) successfully deleted
W20260620 20:16:05.231201 6708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:05.231436 6708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:05.231477 6708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:05.235697 6708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:05.235751 6708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:05.235836 6708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:16:05.240995 6708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:45729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=43273
--tserver_master_addrs=127.24.20.190:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:05.241988 6708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:05.242990 6708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:05.252100 6719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:05.252126 6718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:05.252112 6721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:05.252380 6708 server_base.cc:1034] running on GCE node
I20260620 20:16:05.252910 6708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:05.253510 6708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:05.254693 6708 hybrid_clock.cc:648] HybridClock initialized: now 1781986565254658 us; error 43 us; skew 500 ppm
Jun 20 20:16:05 dist-test-slave-pflr krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986565, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:05.257473 6708 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:16:05.258293 6708 webserver.cc:460] Webserver started at http://127.24.20.130:43273/ using document root <none> and password file <none>
I20260620 20:16:05.258814 6708 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:05.258942 6708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:05.262701 6708 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 20:16:05.264793 6728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:05.265908 6708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:05.266036 6708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-pflr"
server_key: "0502500b98e0a82f2865658b7ea163c9"
server_key_iv: "2cb510680ffbcc1ad3c49043a34a5adc"
server_key_version: "encryptionkey@0"
I20260620 20:16:05.266535 6708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:05.275619 6708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:05.278280 6708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:05.278469 6708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:05.279021 6708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:05.279991 6735 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 20:16:05.283774 6708 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 20:16:05.283867 6708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 20:16:05.283946 6708 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 20:16:05.285562 6708 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 20:16:05.285631 6708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260620 20:16:05.285845 6735 tablet_bootstrap.cc:492] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Bootstrap starting.
I20260620 20:16:05.291162 6735 log.cc:824] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:05.295897 6708 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:45729
I20260620 20:16:05.295969 6843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:45729 every 8 connection(s)
I20260620 20:16:05.296846 6708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:05.300350 6735 tablet_bootstrap.cc:492] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:16:05.300462 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 6708
I20260620 20:16:05.300807 6735 tablet_bootstrap.cc:492] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Bootstrap complete.
I20260620 20:16:05.300913 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:35197
--local_ip_for_outbound_sockets=127.24.20.131
--tserver_master_addrs=127.24.20.190:40511
--webserver_port=40531
--webserver_interface=127.24.20.131
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:16:05.301214 6735 ts_tablet_manager.cc:1397] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20260620 20:16:05.303987 6735 raft_consensus.cc:348] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } }
I20260620 20:16:05.304392 6735 raft_consensus.cc:729] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3b718d1ec14a2ca2ee1fc0e47ba9b4, State: Initialized, Role: FOLLOWER
I20260620 20:16:05.304936 6735 consensus_queue.cc:260] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } }
I20260620 20:16:05.305981 6735 ts_tablet_manager.cc:1428] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
Jun 20 20:16:05 dist-test-slave-pflr krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986565, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:05.312472 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.298676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:33635 (local address 127.24.20.190:40511)
0620 20:16:05.298811 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:05.298815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:05.299405 (+ 590us) server_negotiation.cc:407] Connection header received
0620 20:16:05.300406 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:05.300409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:05.300466 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:05.300558 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:05.302137 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.302970 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.304790 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.305148 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.305260 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.308792 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:05.308794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:05.308797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:05.308820 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:05.310246 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.310876 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.310879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.310880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.310950 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.311345 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.311347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.311348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.311511 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:05.311586 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:05.312243 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 20:16:05.312354 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:16:05.313484 6846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.298796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:33635)
0620 20:16:05.299261 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:05.299299 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:05.300131 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:05.300766 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:05.300775 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:05.301264 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:05.301962 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.301984 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.303142 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.303149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.304586 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.304631 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.305388 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.305392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.305571 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.306751 (+ 1180us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:05.306767 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:05.308617 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:05.310364 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:05.310373 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:05.310390 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:05.310770 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:05.311046 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:05.311051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:05.311057 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:05.311244 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:05.311614 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:05.311623 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:05.311950 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 20:16:05.312182 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 20:16:05.312535 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":145,"threads_started":1}
I20260620 20:16:05.314698 6844 heartbeater.cc:347] Connected to a master server at 127.24.20.190:40511
I20260620 20:16:05.314930 6844 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:05.315402 6844 heartbeater.cc:511] Master 127.24.20.190:40511 requested a full tablet report, sending...
I20260620 20:16:05.316998 6516 ts_manager.cc:194] Registered new tserver with Master: df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729)
I20260620 20:16:05.318146 6516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:33635
I20260620 20:16:05.320089 6844 heartbeater.cc:503] Master 127.24.20.190:40511 was elected leader, sending a full tablet report...
Jun 20 20:16:05 dist-test-slave-pflr krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986564, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.130@KRBTEST.COM
I20260620 20:16:05.327179 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.317846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:45729 (local address 127.0.0.1:42190)
0620 20:16:05.318006 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:05.318019 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:05.318098 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:05.318819 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:05.318822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:05.319005 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:05.319101 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.319108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.320482 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.320486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.321090 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.321100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.321825 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.321829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.321922 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.322427 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:05.322438 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:05.323771 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:05.325633 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:05.325640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:05.325642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:05.325837 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:05.326141 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:05.326144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:05.326145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:05.326226 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:05.326662 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:05.326665 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:05.326757 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 20:16:05.326877 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 20:16:05.327010 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 20:16:05.327271 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.317963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.24.20.130:45729)
0620 20:16:05.318304 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:05.318318 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:05.318338 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:16:05.318401 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:05.318407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:05.318628 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:05.318820 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:05.319223 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.320376 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.321213 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.321702 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.321827 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.323904 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:05.323912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:05.323924 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:05.323953 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:05.325512 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.325950 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.325956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.325957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.326038 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.326326 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.326329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.326331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.326545 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:05.326692 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:05.326957 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 20:16:05.327109 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
I20260620 20:16:05.328236 6777 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d61a04afbe47de9516068f16ba4d7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:04 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:42190
I20260620 20:16:05.329159 6850 tablet_replica.cc:330] stopping tablet replica
I20260620 20:16:05.329452 6850 raft_consensus.cc:2229] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:05.329686 6850 raft_consensus.cc:2258] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:05.330724 6850 ts_tablet_manager.cc:1905] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:05.335053 6850 ts_tablet_manager.cc:1918] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:05.335146 6850 log.cc:1196] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/a3d61a04afbe47de9516068f16ba4d7d
I20260620 20:16:05.335421 6850 ts_tablet_manager.cc:1939] T a3d61a04afbe47de9516068f16ba4d7d P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Deleting consensus metadata
I20260620 20:16:05.337100 6502 catalog_manager.cc:4841] TS df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729): tablet a3d61a04afbe47de9516068f16ba4d7d (table test-table [id=6a4605007bf349a1b34f414e52d57c32]) successfully deleted
W20260620 20:16:05.415850 6847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:05.416085 6847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:05.416126 6847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:05.419965 6847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:05.420013 6847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:05.420068 6847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:16:05.424988 6847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:35197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=40531
--tserver_master_addrs=127.24.20.190:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:05.425869 6847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:05.426772 6847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:05.436296 6855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:05.436376 6858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:05.436455 6847 server_base.cc:1034] running on GCE node
W20260620 20:16:05.436558 6856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:05.436949 6847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:05.437512 6847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:05.438704 6847 hybrid_clock.cc:648] HybridClock initialized: now 1781986565438684 us; error 27 us; skew 500 ppm
Jun 20 20:16:05 dist-test-slave-pflr krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986565, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:05.441442 6847 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:16:05.442263 6847 webserver.cc:460] Webserver started at http://127.24.20.131:40531/ using document root <none> and password file <none>
I20260620 20:16:05.442791 6847 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:05.442863 6847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:05.446420 6847 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 20:16:05.448310 6865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:05.449579 6847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:05.449702 6847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "c2370e07aa23459da0ffbf0ae47d72d6"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-pflr"
server_key: "825463e52c44bce65ea7a84fbbfeadf5"
server_key_iv: "8bea81ecff9ca4b2f75a0c5c895c2052"
server_key_version: "encryptionkey@0"
I20260620 20:16:05.450206 6847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:05.464774 6847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:05.467993 6847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:05.468191 6847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:05.468833 6847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:05.469693 6847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:05.469772 6847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:05.469825 6847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:05.469841 6847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:05.478484 6847 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:35197
I20260620 20:16:05.478554 6978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:35197 every 8 connection(s)
I20260620 20:16:05.479625 6847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:05.487625 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 6847
Jun 20 20:16:05 dist-test-slave-pflr krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986565, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:05.493650 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.481562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:42759 (local address 127.24.20.190:40511)
0620 20:16:05.481705 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:05.481710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:05.482541 (+ 831us) server_negotiation.cc:407] Connection header received
0620 20:16:05.483895 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:05.483898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:05.483958 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:05.484065 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:05.485216 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.485754 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.486860 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.487092 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.487205 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.490102 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:05.490107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:05.490114 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:05.490136 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:05.491596 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.492114 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.492119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.492121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.492201 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.492534 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.492539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.492541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.492767 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:05.492873 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:05.493362 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 20:16:05.493490 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:16:05.494576 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.481961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:42759)
0620 20:16:05.482397 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:05.482436 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:05.483688 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:05.484232 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:05.484238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:05.484567 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:05.485078 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.485091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.485875 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.485878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.486737 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.486746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.487237 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.487241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.487405 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.488049 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:05.488064 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:05.489928 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:05.491730 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:05.491735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:05.491748 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:05.492010 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:05.492292 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:05.492296 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:05.492298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:05.492438 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:05.492872 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:05.492878 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:05.493114 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 20:16:05.493295 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 20:16:05.493672 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260620 20:16:05.495954 6979 heartbeater.cc:347] Connected to a master server at 127.24.20.190:40511
I20260620 20:16:05.496235 6979 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:05.496501 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.488287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58848 (local address 127.24.20.190:40511)
0620 20:16:05.488530 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:05.488536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:05.488651 (+ 115us) server_negotiation.cc:407] Connection header received
0620 20:16:05.488806 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:05.488809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:05.488851 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:05.488937 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:05.489860 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.490367 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.491230 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.491472 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.491584 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.492984 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:05.492987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:05.492989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:05.493013 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:05.494691 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.495175 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.495179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.495181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.495266 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:05.495579 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:05.495582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:05.495583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:05.495776 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:05.496002 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:05.496194 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 20:16:05.496334 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260620 20:16:05.496827 6979 heartbeater.cc:511] Master 127.24.20.190:40511 requested a full tablet report, sending...
I20260620 20:16:05.497833 6516 ts_manager.cc:194] Registered new tserver with Master: c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197)
I20260620 20:16:05.498423 6516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:42759
I20260620 20:16:05.499080 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:16:05.503266 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.499702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.24.20.190:40511)
0620 20:16:05.499827 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:05.499832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:05.499980 (+ 148us) server_negotiation.cc:407] Connection header received
0620 20:16:05.500082 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:05.500085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:05.500193 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:05.500300 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:05.500491 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.501149 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.501994 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.502263 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.502351 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.502634 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:05.502973 (+ 339us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:05.503065 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:05.503120 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 20:16:05.503168 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:16:05.503842 6515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:16:05.506928 6515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:16:05.513787 6777 tablet_service.cc:1467] Processing CreateTablet for tablet 1668e9040ced44de9e7e5b0312f868cb (DEFAULT_TABLE table=test-table [id=216b323e60f54550871c12e62b619589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:05.514055 6629 tablet_service.cc:1467] Processing CreateTablet for tablet 1668e9040ced44de9e7e5b0312f868cb (DEFAULT_TABLE table=test-table [id=216b323e60f54550871c12e62b619589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:05.514374 6777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1668e9040ced44de9e7e5b0312f868cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:05.514596 6629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1668e9040ced44de9e7e5b0312f868cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:05.517026 6735 tablet_bootstrap.cc:492] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Bootstrap starting.
I20260620 20:16:05.517050 6596 tablet_bootstrap.cc:492] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767: Bootstrap starting.
I20260620 20:16:05.517822 6735 tablet_bootstrap.cc:654] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:05.517874 6596 tablet_bootstrap.cc:654] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:05.519446 6735 tablet_bootstrap.cc:492] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4: No bootstrap required, opened a new log
I20260620 20:16:05.519550 6735 ts_tablet_manager.cc:1397] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:05.519593 6596 tablet_bootstrap.cc:492] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767: No bootstrap required, opened a new log
I20260620 20:16:05.519672 6596 ts_tablet_manager.cc:1397] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 20:16:05.519984 6735 raft_consensus.cc:348] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:05.520112 6735 raft_consensus.cc:374] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:05.520074 6596 raft_consensus.cc:348] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:05.520159 6735 raft_consensus.cc:729] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df3b718d1ec14a2ca2ee1fc0e47ba9b4, State: Initialized, Role: FOLLOWER
I20260620 20:16:05.520190 6596 raft_consensus.cc:374] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:05.520237 6596 raft_consensus.cc:729] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c684e692bb4315a7b0134b14028767, State: Initialized, Role: FOLLOWER
I20260620 20:16:05.520284 6735 consensus_queue.cc:260] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:05.520372 6596 consensus_queue.cc:260] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:05.520690 6596 ts_tablet_manager.cc:1428] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:16:05.521098 6735 ts_tablet_manager.cc:1428] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:16:05.521586 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.513058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:35197 (local address 127.0.0.1:51136)
0620 20:16:05.513212 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:05.513226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:05.513304 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:05.517055 (+ 3751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:05.517061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:05.517084 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:05.517182 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.517189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.518906 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.518909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.520029 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.520039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.521178 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.521181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.521276 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.521326 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 20:16:05.521380 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 20:16:05.521458 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 20:16:05.522392 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.513190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.24.20.131:35197)
0620 20:16:05.513553 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:05.513567 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:05.513589 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:16:05.513656 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:05.513661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:05.516862 (+ 3201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:05.517335 (+ 473us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:05.517480 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.518757 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.520190 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.521054 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.521894 (+ 840us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.522079 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:05.522133 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 20:16:05.522234 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":165,"threads_started":1}
I20260620 20:16:05.525271 6913 tablet_service.cc:1467] Processing CreateTablet for tablet 1668e9040ced44de9e7e5b0312f868cb (DEFAULT_TABLE table=test-table [id=216b323e60f54550871c12e62b619589]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:05.526298 6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1668e9040ced44de9e7e5b0312f868cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:05.533005 6991 tablet_bootstrap.cc:492] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6: Bootstrap starting.
I20260620 20:16:05.535255 6991 tablet_bootstrap.cc:654] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:05.536388 6991 log.cc:824] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:05.538341 6991 tablet_bootstrap.cc:492] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6: No bootstrap required, opened a new log
I20260620 20:16:05.538560 6991 ts_tablet_manager.cc:1397] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 20:16:05.541479 6991 raft_consensus.cc:348] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:05.541718 6991 raft_consensus.cc:374] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:05.541795 6991 raft_consensus.cc:729] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2370e07aa23459da0ffbf0ae47d72d6, State: Initialized, Role: FOLLOWER
I20260620 20:16:05.542230 6991 consensus_queue.cc:260] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:05.542948 6979 heartbeater.cc:503] Master 127.24.20.190:40511 was elected leader, sending a full tablet report...
I20260620 20:16:05.543256 6991 ts_tablet_manager.cc:1428] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 20:16:05.547441 6993 raft_consensus.cc:482] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:05.547603 6993 raft_consensus.cc:504] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
W20260620 20:16:05.547799 6845 tablet.cc:2369] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:16:05.548784 6993 leader_election.cc:288] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729), f0c684e692bb4315a7b0134b14028767 (127.24.20.129:38101)
I20260620 20:16:05.553735 6712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.549146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:42187 (local address 127.24.20.129:38101)
0620 20:16:05.549323 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:05.549330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:05.549348 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:16:05.549419 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:05.549423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:05.549484 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:05.549598 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:05.549901 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.550738 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.552242 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.552947 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.553071 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.553210 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:05.553518 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 20:16:05.553594 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:16:05.553735 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.549171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:39517 (local address 127.24.20.130:45729)
0620 20:16:05.549299 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:05.549304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:05.549619 (+ 315us) server_negotiation.cc:407] Connection header received
0620 20:16:05.549737 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:05.549739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:05.549784 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:05.549877 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:05.550104 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.550883 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.552309 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.552957 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.553071 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.553210 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:05.553526 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 20:16:05.553597 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:16:05.553731 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.548951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:38101 (local address 127.24.20.131:42187)
0620 20:16:05.549108 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:05.549123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:05.549207 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:05.549627 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:05.549629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:05.549654 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:05.549789 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.549795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.550844 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.550846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.552092 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.552103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.553074 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.553078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.553256 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.553386 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 20:16:05.553470 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 20:16:05.553603 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":102,"spinlock_wait_cycles":3328}
I20260620 20:16:05.553743 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.548950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:45729 (local address 127.24.20.131:39517)
0620 20:16:05.549540 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:05.549553 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:05.549651 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:05.549897 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:05.549900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:05.549916 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:05.550009 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.550013 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.550981 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.550984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.552166 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:05.552176 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.553074 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.553078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:05.553257 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.553386 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 20:16:05.553470 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 20:16:05.553583 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":392,"spinlock_wait_cycles":1280,"thread_start_us":95,"threads_started":1}
I20260620 20:16:05.554553 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1668e9040ced44de9e7e5b0312f868cb" candidate_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" is_pre_election: true
I20260620 20:16:05.554690 6646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1668e9040ced44de9e7e5b0312f868cb" candidate_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0c684e692bb4315a7b0134b14028767" is_pre_election: true
I20260620 20:16:05.554868 6798 raft_consensus.cc:2454] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2370e07aa23459da0ffbf0ae47d72d6 in term 0.
I20260620 20:16:05.555008 6646 raft_consensus.cc:2454] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2370e07aa23459da0ffbf0ae47d72d6 in term 0.
I20260620 20:16:05.555337 6868 leader_election.cc:302] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2370e07aa23459da0ffbf0ae47d72d6, df3b718d1ec14a2ca2ee1fc0e47ba9b4; no voters:
I20260620 20:16:05.555590 6993 raft_consensus.cc:2781] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:16:05.555682 6993 raft_consensus.cc:482] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:16:05.555740 6993 raft_consensus.cc:3037] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:05.556747 6993 raft_consensus.cc:504] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:05.557075 6993 leader_election.cc:288] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [CANDIDATE]: Term 1 election: Requested vote from peers df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729), f0c684e692bb4315a7b0134b14028767 (127.24.20.129:38101)
I20260620 20:16:05.557405 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1668e9040ced44de9e7e5b0312f868cb" candidate_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4"
I20260620 20:16:05.557528 6798 raft_consensus.cc:3037] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:05.557601 6646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1668e9040ced44de9e7e5b0312f868cb" candidate_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0c684e692bb4315a7b0134b14028767"
I20260620 20:16:05.557708 6646 raft_consensus.cc:3037] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:05.558435 6798 raft_consensus.cc:2454] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2370e07aa23459da0ffbf0ae47d72d6 in term 1.
I20260620 20:16:05.558722 6646 raft_consensus.cc:2454] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2370e07aa23459da0ffbf0ae47d72d6 in term 1.
I20260620 20:16:05.558714 6868 leader_election.cc:302] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2370e07aa23459da0ffbf0ae47d72d6, df3b718d1ec14a2ca2ee1fc0e47ba9b4; no voters:
I20260620 20:16:05.558918 6993 raft_consensus.cc:2781] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:05.559155 6993 raft_consensus.cc:686] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [term 1 LEADER]: Becoming Leader. State: Replica: c2370e07aa23459da0ffbf0ae47d72d6, State: Running, Role: LEADER
I20260620 20:16:05.559458 6993 consensus_queue.cc:237] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:05.562583 6516 catalog_manager.cc:5495] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131). New cstate: current_term: 1 leader_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } health_report { overall_health: UNKNOWN } } }
I20260620 20:16:05.585090 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:05.581013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.24.20.131:35197)
0620 20:16:05.581162 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:05.581169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:05.581256 (+ 87us) server_negotiation.cc:407] Connection header received
0620 20:16:05.581538 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:05.581542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:05.581598 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:05.581721 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:05.581981 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.582546 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.583479 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:05.583719 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:05.583839 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:05.584177 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:05.584681 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:05.584823 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:05.584878 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 20:16:05.584935 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:16:05.590891 6798 raft_consensus.cc:1261] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 1 FOLLOWER]: Refusing update from remote peer c2370e07aa23459da0ffbf0ae47d72d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:05.590891 6646 raft_consensus.cc:1261] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Refusing update from remote peer c2370e07aa23459da0ffbf0ae47d72d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:05.591565 6995 consensus_queue.cc:1035] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:05.591799 6993 consensus_queue.cc:1035] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:05.596678 6998 mvcc.cc:205] Tried to move back new op lower bound from 7299016972653805568 to 7299016972533956608. Current Snapshot: MvccSnapshot[applied={T|T < 7299016972653805568}]
I20260620 20:16:05.600145 7000 mvcc.cc:205] Tried to move back new op lower bound from 7299016972653805568 to 7299016972533956608. Current Snapshot: MvccSnapshot[applied={T|T < 7299016972653805568 or (T in {7299016972653805568})}]
I20260620 20:16:05.605952 6516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:05.606114 6516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:05.608415 6516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Sending DeleteTablet for 3 replicas of tablet 1668e9040ced44de9e7e5b0312f868cb
I20260620 20:16:05.609150 6777 tablet_service.cc:1514] Processing DeleteTablet for tablet 1668e9040ced44de9e7e5b0312f868cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:05 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:42190
I20260620 20:16:05.609126 6913 tablet_service.cc:1514] Processing DeleteTablet for tablet 1668e9040ced44de9e7e5b0312f868cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:05 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:51136
I20260620 20:16:05.609450 6629 tablet_service.cc:1514] Processing DeleteTablet for tablet 1668e9040ced44de9e7e5b0312f868cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:05 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:56678
I20260620 20:16:05.609545 6850 tablet_replica.cc:330] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4: stopping tablet replica
I20260620 20:16:05.609622 6713 tablet_replica.cc:330] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767: stopping tablet replica
I20260620 20:16:05.609673 6850 raft_consensus.cc:2229] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:05.609738 6713 raft_consensus.cc:2229] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:05.609752 7007 tablet_replica.cc:330] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6: stopping tablet replica
I20260620 20:16:05.609813 6850 raft_consensus.cc:2258] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:05.609862 6713 raft_consensus.cc:2258] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:05.610060 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 6554
I20260620 20:16:05.610049 7007 raft_consensus.cc:2229] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:16:05.610383 7007 raft_consensus.cc:2258] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:05.611822 7007 ts_tablet_manager.cc:1905] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:05.614427 7007 ts_tablet_manager.cc:1918] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:05.614548 7007 log.cc:1196] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/1668e9040ced44de9e7e5b0312f868cb
I20260620 20:16:05.614857 7007 ts_tablet_manager.cc:1939] T 1668e9040ced44de9e7e5b0312f868cb P c2370e07aa23459da0ffbf0ae47d72d6: Deleting consensus metadata
I20260620 20:16:05.615859 6500 catalog_manager.cc:4841] TS c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197): tablet 1668e9040ced44de9e7e5b0312f868cb (table test-table [id=216b323e60f54550871c12e62b619589]) successfully deleted
I20260620 20:16:05.617272 6850 ts_tablet_manager.cc:1905] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 20:16:05.618484 6503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:38101 (error 108)
I20260620 20:16:05.619382 6850 ts_tablet_manager.cc:1918] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:05.619475 6850 log.cc:1196] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/1668e9040ced44de9e7e5b0312f868cb
I20260620 20:16:05.619704 6850 ts_tablet_manager.cc:1939] T 1668e9040ced44de9e7e5b0312f868cb P df3b718d1ec14a2ca2ee1fc0e47ba9b4: Deleting consensus metadata
W20260620 20:16:05.619946 6711 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:05.619407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:38101 (local address 127.0.0.1:56682)
0620 20:16:05.619573 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:05.619588 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:05.619669 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:05.619846 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:38101: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 20:16:05.620362 6502 catalog_manager.cc:4841] TS df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729): tablet 1668e9040ced44de9e7e5b0312f868cb (table test-table [id=216b323e60f54550871c12e62b619589]) successfully deleted
W20260620 20:16:05.620114 6503 catalog_manager.cc:4584] TS f0c684e692bb4315a7b0134b14028767 (127.24.20.129:38101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1668e9040ced44de9e7e5b0312f868cb: Network error: Client connection negotiation failed: client connection to 127.24.20.129:38101: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:16:05.621006 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 6708
I20260620 20:16:05.627717 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 6847
I20260620 20:16:05.632469 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 6483
I20260620 20:16:05.638398 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:40511
--webserver_interface=127.24.20.190
--webserver_port=43539
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:40511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:05.748692 7009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:05.748975 7009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:05.749046 7009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:05.752969 7009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:05.753053 7009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:05.753091 7009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:05.753150 7009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:05.753189 7009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:05.758492 7009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:40511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=43539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:05.759488 7009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:05.760475 7009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:05.768885 7017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:05.768868 7014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:05.768858 7015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:05.769249 7009 server_base.cc:1034] running on GCE node
I20260620 20:16:05.769842 7009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:05.770989 7009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:05.772173 7009 hybrid_clock.cc:648] HybridClock initialized: now 1781986565772159 us; error 28 us; skew 500 ppm
Jun 20 20:16:05 dist-test-slave-pflr krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986565, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:05.774973 7009 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:16:05.775889 7009 webserver.cc:460] Webserver started at http://127.24.20.190:43539/ using document root <none> and password file <none>
I20260620 20:16:05.776476 7009 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:05.776571 7009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:05.780365 7009 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:16:05.782835 7024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:05.784096 7009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:05.784255 7009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "fd14c2464f7b40abbdeb3c9db5575371"
format_stamp: "Formatted at 2026-06-20 20:16:03 on dist-test-slave-pflr"
server_key: "8593b1ba10690651e7f1cd84602ac9ca"
server_key_iv: "bcdc59d1db7e4fddc50b63e25685e9ca"
server_key_version: "encryptionkey@0"
I20260620 20:16:05.784878 7009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:05.824118 7009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:05.827723 7009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:05.827940 7009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:05.835073 7009 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:40511
I20260620 20:16:05.835476 7076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:40511 every 8 connection(s)
I20260620 20:16:05.836205 7009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:05.840387 7077 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:16:05.842239 7077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Bootstrap starting.
I20260620 20:16:05.845086 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 7009
I20260620 20:16:05.845556 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:38101
--local_ip_for_outbound_sockets=127.24.20.129
--tserver_master_addrs=127.24.20.190:40511
--webserver_port=36275
--webserver_interface=127.24.20.129
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:16:05.859125 7077 log.cc:824] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:05.869951 7077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:16:05.870311 7077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371: Bootstrap complete.
I20260620 20:16:05.874238 7077 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } }
I20260620 20:16:05.874627 7077 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd14c2464f7b40abbdeb3c9db5575371, State: Initialized, Role: FOLLOWER
I20260620 20:16:05.875106 7077 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } }
I20260620 20:16:05.875248 7077 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:05.875335 7077 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:05.875454 7077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 2 FOLLOWER]: Advancing to term 3
I20260620 20:16:05.877302 7077 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } }
I20260620 20:16:05.877626 7077 leader_election.cc:302] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd14c2464f7b40abbdeb3c9db5575371; no voters:
I20260620 20:16:05.877923 7077 leader_election.cc:288] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 20:16:05.878052 7081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 20:16:05.878310 7081 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [term 3 LEADER]: Becoming Leader. State: Replica: fd14c2464f7b40abbdeb3c9db5575371, State: Running, Role: LEADER
I20260620 20:16:05.878615 7081 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } }
I20260620 20:16:05.879119 7077 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:05.880014 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fd14c2464f7b40abbdeb3c9db5575371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } } }
I20260620 20:16:05.880136 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:05.880337 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd14c2464f7b40abbdeb3c9db5575371. Latest consensus state: current_term: 3 leader_uuid: "fd14c2464f7b40abbdeb3c9db5575371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd14c2464f7b40abbdeb3c9db5575371" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40511 } } }
I20260620 20:16:05.880414 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd14c2464f7b40abbdeb3c9db5575371 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:05.881162 7090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:05.886482 7090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:05.887063 7090 catalog_manager.cc:1259] Loaded cluster ID: fdf419a5dc4148c9a3f00277e8aa69ec
I20260620 20:16:05.887121 7090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 20:16:05.888055 7090 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd62b10e9ad google::LogMessage::Fail() at ??:0
@ 0x7fd62b11188a google::LogMessage::SendToLog() at ??:0
@ 0x7fd62b10e4de google::LogMessage::Flush() at ??:0
@ 0x7fd62b10fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd635a4dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd635a4ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd635a4b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd635a8f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd62d3603d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd62d360c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd62d3623f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd62d351512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd62e9956db start_thread at ??:0
@ 0x7fd629c0c61f clone at ??:0
W20260620 20:16:05.984761 7079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:05.985026 7079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:05.985071 7079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:05.989218 7079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:05.989288 7079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:05.989349 7079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:16:05.994736 7079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:38101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=36275
--tserver_master_addrs=127.24.20.190:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:05.995814 7079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:05.996924 7079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:06.007006 7119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:06.007095 7079 server_base.cc:1034] running on GCE node
W20260620 20:16:06.007016 7118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:06.007323 7121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:06.007866 7079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:06.008684 7079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:06.009879 7079 hybrid_clock.cc:648] HybridClock initialized: now 1781986566009844 us; error 45 us; skew 500 ppm
Jun 20 20:16:06 dist-test-slave-pflr krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986566, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:06.013286 7079 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:16:06.014139 7079 webserver.cc:460] Webserver started at http://127.24.20.129:36275/ using document root <none> and password file <none>
I20260620 20:16:06.014717 7079 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:06.014777 7079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:06.018865 7079 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:06.021194 7128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:06.022562 7079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:16:06.022706 7079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "f0c684e692bb4315a7b0134b14028767"
format_stamp: "Formatted at 2026-06-20 20:16:03 on dist-test-slave-pflr"
server_key: "539cfb23ceb1fe980e94516f94c26474"
server_key_iv: "13177ea81855668cd7aaeebb156ebf76"
server_key_version: "encryptionkey@0"
I20260620 20:16:06.023253 7079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:06.035290 7079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:06.038576 7079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:06.038776 7079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:06.039350 7079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:06.040422 7135 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 20:16:06.046319 7079 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 20:16:06.046415 7079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 20:16:06.046566 7079 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 20:16:06.048156 7079 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 20:16:06.048228 7079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 20:16:06.048563 7135 tablet_bootstrap.cc:492] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767: Bootstrap starting.
I20260620 20:16:06.055538 7135 log.cc:824] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:06.058734 7079 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:38101
I20260620 20:16:06.058914 7243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:38101 every 8 connection(s)
I20260620 20:16:06.059721 7079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:06.062516 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 7079
I20260620 20:16:06.062973 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:45729
--local_ip_for_outbound_sockets=127.24.20.130
--tserver_master_addrs=127.24.20.190:40511
--webserver_port=43273
--webserver_interface=127.24.20.130
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:16:06.065256 7135 tablet_bootstrap.cc:492] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:16:06.065697 7135 tablet_bootstrap.cc:492] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767: Bootstrap complete.
I20260620 20:16:06.066066 7135 ts_tablet_manager.cc:1397] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20260620 20:16:06.069097 7135 raft_consensus.cc:348] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:06.069691 7135 raft_consensus.cc:729] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c684e692bb4315a7b0134b14028767, State: Initialized, Role: FOLLOWER
I20260620 20:16:06.070274 7135 consensus_queue.cc:260] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:06.071352 7135 ts_tablet_manager.cc:1428] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 20:16:06.122219 7246 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:16:06.061906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:43443)
0620 20:16:06.062475 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:06.062536 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:06.064129 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:06.121216 (+ 57087us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":117,"threads_started":1}
W20260620 20:16:06.122833 7244 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:40511 (0 consecutive failures): Network error: Failed to ping master at 127.24.20.190:40511: Client connection negotiation failed: client connection to 127.24.20.190:40511: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:16:06.123625 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:06.123324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:50765)
0620 20:16:06.123467 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:06.123549 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 20:16:06.124325 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:06.124091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:35385)
0620 20:16:06.124218 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:06.124268 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 20:16:06.124490 7244 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:16:06.176421 7247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:06.176753 7247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:06.176801 7247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:06.180811 7247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:06.180868 7247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:06.180927 7247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:16:06.186177 7247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:45729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=43273
--tserver_master_addrs=127.24.20.190:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:06.187177 7247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:06.188159 7247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:06.197901 7254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:06.197896 7253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:06.197952 7247 server_base.cc:1034] running on GCE node
W20260620 20:16:06.197904 7256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:06.198643 7247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:06.199304 7247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:06.200461 7247 hybrid_clock.cc:648] HybridClock initialized: now 1781986566200424 us; error 51 us; skew 500 ppm
Jun 20 20:16:06 dist-test-slave-pflr krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986566, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:06.203390 7247 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:16:06.204308 7247 webserver.cc:460] Webserver started at http://127.24.20.130:43273/ using document root <none> and password file <none>
I20260620 20:16:06.204924 7247 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:06.204988 7247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:06.208801 7247 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 20:16:06.210774 7263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:06.211997 7247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:06.212147 7247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-pflr"
server_key: "0502500b98e0a82f2865658b7ea163c9"
server_key_iv: "2cb510680ffbcc1ad3c49043a34a5adc"
server_key_version: "encryptionkey@0"
I20260620 20:16:06.212738 7247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:06.225128 7247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:06.228097 7247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:06.228294 7247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:06.228935 7247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:06.229744 7247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:06.229813 7247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:06.229882 7247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:06.229935 7247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:06.238328 7247 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:45729
I20260620 20:16:06.238355 7376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:45729 every 8 connection(s)
I20260620 20:16:06.239274 7247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:06.239490 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 7247
I20260620 20:16:06.239960 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:35197
--local_ip_for_outbound_sockets=127.24.20.131
--tserver_master_addrs=127.24.20.190:40511
--webserver_port=40531
--webserver_interface=127.24.20.131
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:16:06.243755 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:06.241488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:39405)
0620 20:16:06.242016 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:06.242178 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
W20260620 20:16:06.244221 7377 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:40511 (0 consecutive failures): Network error: Failed to ping master at 127.24.20.190:40511: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
I20260620 20:16:06.244849 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:06.244662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:49521)
0620 20:16:06.244755 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:06.244799 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 20:16:06.245471 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:06.245298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:58677)
0620 20:16:06.245388 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:06.245425 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 20:16:06.245651 7377 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:16:06.349675 7379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:06.349906 7379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:06.349947 7379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:06.353914 7379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:06.353963 7379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:06.354017 7379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:16:06.359036 7379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:35197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=40531
--tserver_master_addrs=127.24.20.190:40511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:06.360036 7379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:06.361025 7379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:06.370388 7388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:06.370411 7386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:06.370595 7379 server_base.cc:1034] running on GCE node
W20260620 20:16:06.370402 7385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:06.371022 7379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:06.371609 7379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:06.372773 7379 hybrid_clock.cc:648] HybridClock initialized: now 1781986566372746 us; error 31 us; skew 500 ppm
Jun 20 20:16:06 dist-test-slave-pflr krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986566, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:06.375353 7379 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:16:06.376152 7379 webserver.cc:460] Webserver started at http://127.24.20.131:40531/ using document root <none> and password file <none>
I20260620 20:16:06.376780 7379 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:06.376840 7379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:06.380410 7379 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 20:16:06.382393 7395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:06.383543 7379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:06.383670 7379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "c2370e07aa23459da0ffbf0ae47d72d6"
format_stamp: "Formatted at 2026-06-20 20:16:04 on dist-test-slave-pflr"
server_key: "825463e52c44bce65ea7a84fbbfeadf5"
server_key_iv: "8bea81ecff9ca4b2f75a0c5c895c2052"
server_key_version: "encryptionkey@0"
I20260620 20:16:06.384193 7379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:06.420980 7379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:06.423965 7379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:06.424183 7379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:06.424813 7379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:06.425644 7379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:06.425716 7379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:06.425787 7379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:06.425843 7379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:06.435130 7379 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:35197
I20260620 20:16:06.435158 7508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:35197 every 8 connection(s)
I20260620 20:16:06.436128 7379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:06.437318 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 7379
I20260620 20:16:06.440060 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:06.438330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:57033)
0620 20:16:06.438821 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:06.438928 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":132,"threads_started":1}
W20260620 20:16:06.440478 7509 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:40511 (0 consecutive failures): Network error: Failed to ping master at 127.24.20.190:40511: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
I20260620 20:16:06.441103 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:06.440922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:47905)
0620 20:16:06.441007 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:06.441050 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 20:16:06.441663 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:06.441499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:38309)
0620 20:16:06.441580 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:06.441617 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 20:16:06.441854 7509 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:16:07.125517 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:07.125101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:38785)
0620 20:16:07.125350 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:07.125420 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260620 20:16:07.246569 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:07.246181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:35939)
0620 20:16:07.246412 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:07.246468 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260620 20:16:07.442852 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:07.442474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:38031)
0620 20:16:07.442702 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:07.442759 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20260620 20:16:07.747148 7516 raft_consensus.cc:482] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:07.747367 7516 raft_consensus.cc:504] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:07.748517 7516 leader_election.cc:288] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197), df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729)
Jun 20 20:16:07 dist-test-slave-pflr krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986566, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.131@KRBTEST.COM
Jun 20 20:16:07 dist-test-slave-pflr krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986566, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.130@KRBTEST.COM
I20260620 20:16:07.762816 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:07.748992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:54637 (local address 127.24.20.130:45729)
0620 20:16:07.749387 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:07.749399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:07.749432 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:16:07.750139 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:07.750144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:07.750434 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:07.750723 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:07.752022 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:07.753047 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:07.754206 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:07.754657 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:07.754835 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:07.757929 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:07.757936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:07.757953 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:07.757991 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:07.759984 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:07.760449 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:07.760454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:07.760459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:07.760643 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:07.760964 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:07.760967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:07.760969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:07.761461 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:07.761605 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:07.762327 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 20:16:07.762625 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":78,"threads_started":1}
I20260620 20:16:07.762816 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:07.748930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:59803 (local address 127.24.20.131:35197)
0620 20:16:07.749196 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:07.749223 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:07.749270 (+ 47us) server_negotiation.cc:407] Connection header received
0620 20:16:07.750021 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:07.750028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:07.750392 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:07.750696 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:07.752026 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:07.753048 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:07.754216 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:07.754660 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:07.754839 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:07.757634 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:07.757639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:07.757649 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:07.757675 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:07.759716 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:07.760316 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:07.760321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:07.760325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:07.760480 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:07.760964 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:07.760967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:07.760969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:07.761461 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:07.761599 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:07.762327 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 20:16:07.762622 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":77,"threads_started":1}
I20260620 20:16:07.762825 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:07.748841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:45729 (local address 127.24.20.129:54637)
0620 20:16:07.749329 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:07.749342 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:07.749441 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:07.750768 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:07.750774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:07.751304 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:07.751835 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:07.751849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:07.753206 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:07.753209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:07.754080 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:07.754088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:07.754782 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:07.754786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:07.755017 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:07.755684 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:07.755700 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:07.757744 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:07.760100 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:07.760104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:07.760107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:07.760352 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:07.760737 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:07.760740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:07.760742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:07.760856 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:07.761602 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:07.761608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:07.761979 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 20:16:07.762190 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 20:16:07.762588 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":151,"spinlock_wait_cycles":121088,"thread_start_us":69,"threads_started":1}
I20260620 20:16:07.762825 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:07.748771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:35197 (local address 127.24.20.129:59803)
0620 20:16:07.749103 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:07.749119 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:07.749223 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:07.750768 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:07.750775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:07.751314 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:07.751835 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:07.751850 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:07.753206 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:07.753209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:07.754079 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:07.754088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:07.754782 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:07.754786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:07.755019 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:07.755746 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:07.755758 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:07.757399 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:07.759851 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:07.759856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:07.759865 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:07.760205 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:07.760652 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:07.760655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:07.760657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:07.760856 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:07.761602 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:07.761608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:07.761979 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 20:16:07.762190 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 20:16:07.762638 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":417,"spinlock_wait_cycles":12544,"thread_start_us":86,"threads_started":1}
W20260620 20:16:07.765594 7131 leader_election.cc:341] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729): Not found: Tablet not found: 1668e9040ced44de9e7e5b0312f868cb
W20260620 20:16:07.765774 7129 leader_election.cc:341] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197): Not found: Tablet not found: 1668e9040ced44de9e7e5b0312f868cb
I20260620 20:16:07.765843 7129 leader_election.cc:302] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0c684e692bb4315a7b0134b14028767; no voters: c2370e07aa23459da0ffbf0ae47d72d6, df3b718d1ec14a2ca2ee1fc0e47ba9b4
I20260620 20:16:07.766074 7516 raft_consensus.cc:2726] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:16:08.126817 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:08.126456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:38883)
0620 20:16:08.126656 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:08.126730 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 20:16:08.247740 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:08.247306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:54663)
0620 20:16:08.247574 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:08.247637 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":83,"threads_started":1}
I20260620 20:16:08.443929 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:08.443549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:45991)
0620 20:16:08.443775 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:08.443842 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20260620 20:16:09.127936 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:09.127521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:35723)
0620 20:16:09.127773 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:09.127828 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
I20260620 20:16:09.248827 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:09.248414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:48285)
0620 20:16:09.248687 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:09.248740 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260620 20:16:09.444993 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:09.444575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:43285)
0620 20:16:09.444851 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:09.444902 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":65,"threads_started":1}
I20260620 20:16:09.829231 7526 raft_consensus.cc:482] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:09.829386 7526 raft_consensus.cc:504] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:09.829860 7526 leader_election.cc:288] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197), df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729)
W20260620 20:16:09.830824 7129 leader_election.cc:341] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197): Not found: Tablet not found: 1668e9040ced44de9e7e5b0312f868cb
W20260620 20:16:09.830937 7131 leader_election.cc:341] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729): Not found: Tablet not found: 1668e9040ced44de9e7e5b0312f868cb
I20260620 20:16:09.831012 7131 leader_election.cc:302] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0c684e692bb4315a7b0134b14028767; no voters: c2370e07aa23459da0ffbf0ae47d72d6, df3b718d1ec14a2ca2ee1fc0e47ba9b4
I20260620 20:16:09.831208 7526 raft_consensus.cc:2726] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:16:10.129392 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:10.128999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:41155)
0620 20:16:10.129239 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:10.129296 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260620 20:16:10.249856 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:10.249486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:57595)
0620 20:16:10.249712 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:10.249768 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260620 20:16:10.446002 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:10.445627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:38577)
0620 20:16:10.445859 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:10.445914 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260620 20:16:11.130740 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:11.130291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:50953)
0620 20:16:11.130594 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:11.130647 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":117,"threads_started":1}
I20260620 20:16:11.251127 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:11.250729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:40063)
0620 20:16:11.250980 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:11.251034 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260620 20:16:11.447237 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:11.446859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:56165)
0620 20:16:11.447095 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:11.447149 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260620 20:16:12.131912 7533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:12.131480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:43131)
0620 20:16:12.131756 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:12.131812 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":98,"threads_started":1}
I20260620 20:16:12.252295 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:12.251891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:51983)
0620 20:16:12.252150 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:12.252203 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
I20260620 20:16:12.448376 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:12.447976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:50115)
0620 20:16:12.448235 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:12.448286 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
I20260620 20:16:12.662458 7536 raft_consensus.cc:482] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:12.662657 7536 raft_consensus.cc:504] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:12.663094 7536 leader_election.cc:288] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197), df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729)
W20260620 20:16:12.663873 7131 leader_election.cc:341] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729): Not found: Tablet not found: 1668e9040ced44de9e7e5b0312f868cb
W20260620 20:16:12.663977 7129 leader_election.cc:341] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197): Not found: Tablet not found: 1668e9040ced44de9e7e5b0312f868cb
I20260620 20:16:12.664021 7131 leader_election.cc:302] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0c684e692bb4315a7b0134b14028767; no voters: c2370e07aa23459da0ffbf0ae47d72d6, df3b718d1ec14a2ca2ee1fc0e47ba9b4
I20260620 20:16:12.664179 7536 raft_consensus.cc:2726] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:16:13.133078 7537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:13.132690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:57945)
0620 20:16:13.132921 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:13.132973 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
I20260620 20:16:13.253495 7538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:13.253046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:47225)
0620 20:16:13.253337 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:13.253394 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":73,"threads_started":1}
I20260620 20:16:13.449671 7539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:13.449227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:49335)
0620 20:16:13.449519 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:13.449580 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":80,"threads_started":1}
I20260620 20:16:14.134150 7540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:14.133754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:58447)
0620 20:16:14.133982 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:14.134054 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
I20260620 20:16:14.254642 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:14.254261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:34449)
0620 20:16:14.254497 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:14.254552 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
I20260620 20:16:14.450789 7542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:14.450400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:39397)
0620 20:16:14.450645 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:14.450699 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260620 20:16:15.135257 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:15.134855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:58447)
0620 20:16:15.135111 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:15.135163 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260620 20:16:15.255815 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:15.255391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:37545)
0620 20:16:15.255665 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:15.255724 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
I20260620 20:16:15.452049 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:15.451638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:54243)
0620 20:16:15.451895 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:15.451954 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260620 20:16:16.136432 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:16.136064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:37721)
0620 20:16:16.136287 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:16.136339 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260620 20:16:16.256999 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:16.256572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:53791)
0620 20:16:16.256854 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:16.256907 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":69,"threads_started":1}
I20260620 20:16:16.278908 7548 raft_consensus.cc:482] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:16.279033 7548 raft_consensus.cc:504] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2370e07aa23459da0ffbf0ae47d72d6" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35197 } } peers { permanent_uuid: "df3b718d1ec14a2ca2ee1fc0e47ba9b4" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45729 } } peers { permanent_uuid: "f0c684e692bb4315a7b0134b14028767" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38101 } }
I20260620 20:16:16.279398 7548 leader_election.cc:288] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197), df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729)
W20260620 20:16:16.280102 7129 leader_election.cc:341] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2370e07aa23459da0ffbf0ae47d72d6 (127.24.20.131:35197): Not found: Tablet not found: 1668e9040ced44de9e7e5b0312f868cb
W20260620 20:16:16.280230 7131 leader_election.cc:341] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df3b718d1ec14a2ca2ee1fc0e47ba9b4 (127.24.20.130:45729): Not found: Tablet not found: 1668e9040ced44de9e7e5b0312f868cb
I20260620 20:16:16.280339 7131 leader_election.cc:302] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0c684e692bb4315a7b0134b14028767; no voters: c2370e07aa23459da0ffbf0ae47d72d6, df3b718d1ec14a2ca2ee1fc0e47ba9b4
I20260620 20:16:16.280539 7548 raft_consensus.cc:2726] T 1668e9040ced44de9e7e5b0312f868cb P f0c684e692bb4315a7b0134b14028767 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:16:16.453248 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:16.452861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:36419)
0620 20:16:16.453092 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:16.453154 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260620 20:16:17.137645 7550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:17.137239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:38589)
0620 20:16:17.137489 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:17.137548 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260620 20:16:17.258131 7551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:17.257726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:35081)
0620 20:16:17.257964 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:17.258018 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260620 20:16:17.454349 7552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:17.453939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:42683)
0620 20:16:17.454202 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:17.454260 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
I20260620 20:16:18.138830 7553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.138361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:47611)
0620 20:16:18.138681 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:18.138736 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260620 20:16:18.259225 7554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.258814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:50631)
0620 20:16:18.259073 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:18.259129 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260620 20:16:18.455557 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:18.455129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:42755)
0620 20:16:18.455400 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:18.455462 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":72,"threads_started":1}
I20260620 20:16:19.140023 7556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:19.139609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:41417)
0620 20:16:19.139864 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:19.139923 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260620 20:16:19.260386 7557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:19.259970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:48153)
0620 20:16:19.260240 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:19.260295 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":104,"threads_started":1}
I20260620 20:16:19.456660 7558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:19.456248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:36679)
0620 20:16:19.456472 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:19.456521 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260620 20:16:20.141336 7559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.140829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:42709)
0620 20:16:20.141172 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:20.141234 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":121,"threads_started":1}
I20260620 20:16:20.261500 7560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.261075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:39211)
0620 20:16:20.261314 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:20.261367 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
I20260620 20:16:20.457749 7561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:20.457339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.131:60121)
0620 20:16:20.457602 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:20.457656 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260620 20:16:21.142684 7562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.142241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.129:60683)
0620 20:16:21.142541 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:21.142592 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":82,"threads_started":1}
I20260620 20:16:21.262766 7563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:21.262323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:40511 (local address 127.24.20.130:55257)
0620 20:16:21.262627 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:21.262678 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":118,"threads_started":1}
I20260620 20:16:21.441779 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 7079
I20260620 20:16:21.447515 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 7247
I20260620 20:16:21.453323 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 7379
2026-06-20T20:16:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19105 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:16:21 dist-test-slave-pflr krb5kdc[7566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:16:21 dist-test-slave-pflr krb5kdc[7566](info): set up 2 sockets
Jun 20 20:16:21 dist-test-slave-pflr krb5kdc[7566](info): commencing operation
krb5kdc: starting...
W20260620 20:16:22.493038 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:16:22 dist-test-slave-pflr krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:16:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:16:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:16:22.611687 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:33277
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:33277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:22.718683 7579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:22.718952 7579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:22.718993 7579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:22.722936 7579 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:16:22.722985 7579 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:16:22.723001 7579 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:16:22.723013 7579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:22.723031 7579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:22.723048 7579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:22.723063 7579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:22.723078 7579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:22.728025 7579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:33277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:33277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:22.729013 7579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:22.729934 7579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:22.738016 7587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:22.738015 7585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:22.738066 7584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:22.738385 7579 server_base.cc:1034] running on GCE node
I20260620 20:16:22.738945 7579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:22.740001 7579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:22.741185 7579 hybrid_clock.cc:648] HybridClock initialized: now 1781986582741161 us; error 37 us; skew 500 ppm
Jun 20 20:16:22 dist-test-slave-pflr krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986582, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:22.743937 7579 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:16:22.744827 7579 webserver.cc:460] Webserver started at http://127.24.20.190:34581/ using document root <none> and password file <none>
I20260620 20:16:22.745353 7579 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:22.745441 7579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:22.745692 7579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:22.747630 7579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "e88230dbd74442f6a7f9e53b69001873"
format_stamp: "Formatted at 2026-06-20 20:16:22 on dist-test-slave-pflr"
server_key: "9dd2dd2aae897c3cb90c226b3c700175"
server_key_iv: "7d4a0ab0f0ad299cab4e3c809569753e"
server_key_version: "encryptionkey@0"
I20260620 20:16:22.748162 7579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "e88230dbd74442f6a7f9e53b69001873"
format_stamp: "Formatted at 2026-06-20 20:16:22 on dist-test-slave-pflr"
server_key: "9dd2dd2aae897c3cb90c226b3c700175"
server_key_iv: "7d4a0ab0f0ad299cab4e3c809569753e"
server_key_version: "encryptionkey@0"
I20260620 20:16:22.751672 7579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 20:16:22.753988 7594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:22.755164 7579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:16:22.755298 7579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "e88230dbd74442f6a7f9e53b69001873"
format_stamp: "Formatted at 2026-06-20 20:16:22 on dist-test-slave-pflr"
server_key: "9dd2dd2aae897c3cb90c226b3c700175"
server_key_iv: "7d4a0ab0f0ad299cab4e3c809569753e"
server_key_version: "encryptionkey@0"
I20260620 20:16:22.755401 7579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:22.766422 7579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:22.769260 7579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:22.769450 7579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:22.776101 7579 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:33277
I20260620 20:16:22.776386 7646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:33277 every 8 connection(s)
I20260620 20:16:22.777038 7579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:22.779706 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:22.785104 7647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Bootstrap starting.
I20260620 20:16:22.787262 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 7579
I20260620 20:16:22.787400 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:16:22.787514 7647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:22.787729 24658 external_mini_cluster.cc:1428] Setting key b7f8f70084a3561693260841165a2b5f
I20260620 20:16:22.788393 7647 log.cc:824] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:22.790596 7647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: No bootstrap required, opened a new log
I20260620 20:16:22.794833 7647 raft_consensus.cc:348] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } }
I20260620 20:16:22.795131 7647 raft_consensus.cc:374] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:22.795184 7647 raft_consensus.cc:729] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e88230dbd74442f6a7f9e53b69001873, State: Initialized, Role: FOLLOWER
I20260620 20:16:22.795616 7647 consensus_queue.cc:260] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } }
I20260620 20:16:22.795742 7647 raft_consensus.cc:388] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:22.795835 7647 raft_consensus.cc:482] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:22.795951 7647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:22.797013 7647 raft_consensus.cc:504] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } }
Jun 20 20:16:22 dist-test-slave-pflr krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:22.797384 7647 leader_election.cc:302] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e88230dbd74442f6a7f9e53b69001873; no voters:
I20260620 20:16:22.797739 7647 leader_election.cc:288] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:16:22.797851 7652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:22.798096 7652 raft_consensus.cc:686] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 1 LEADER]: Becoming Leader. State: Replica: e88230dbd74442f6a7f9e53b69001873, State: Running, Role: LEADER
I20260620 20:16:22.798441 7652 consensus_queue.cc:237] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } }
I20260620 20:16:22.799127 7647 sys_catalog.cc:564] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:22.799718 7653 sys_catalog.cc:455] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e88230dbd74442f6a7f9e53b69001873. Latest consensus state: current_term: 1 leader_uuid: "e88230dbd74442f6a7f9e53b69001873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } } }
I20260620 20:16:22.799935 7653 sys_catalog.cc:458] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:22.800155 7654 sys_catalog.cc:455] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e88230dbd74442f6a7f9e53b69001873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } } }
I20260620 20:16:22.800257 7654 sys_catalog.cc:458] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:22.800402 7661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:22.803488 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:22.788808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.24.20.190:33277)
0620 20:16:22.789172 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:22.789214 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:22.789254 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:16:22.790049 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:22.790072 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:22.790446 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:22.790792 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:22.791521 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:22.793020 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:22.793913 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:22.794606 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:22.794763 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:22.797752 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:22.797760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:22.797776 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:22.797809 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:22.799990 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:22.800550 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:22.800556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:22.800562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:22.800727 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:22.801096 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:22.801102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:22.801107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:22.801619 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:22.801841 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:22.802297 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 20:16:22.802595 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
I20260620 20:16:22.803499 7661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:22.809337 7661 catalog_manager.cc:1347] Generated new cluster ID: eed7674130714db094826ae2c66bbd30
I20260620 20:16:22.809412 7661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:22.827920 7661 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:16:22.828860 7661 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:22.833364 7661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Generated new TSK 0
I20260620 20:16:22.834044 7661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:16:22.865255 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:33277
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:22.972963 7673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:22.973187 7673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:22.973227 7673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:22.977051 7673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:22.977099 7673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:22.977152 7673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:16:22.982111 7673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:22.982988 7673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:22.983866 7673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:22.992921 7678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:22.992965 7679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:22.993134 7673 server_base.cc:1034] running on GCE node
W20260620 20:16:22.993260 7681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:22.993638 7673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:22.994287 7673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:22.995481 7673 hybrid_clock.cc:648] HybridClock initialized: now 1781986582995466 us; error 31 us; skew 500 ppm
Jun 20 20:16:22 dist-test-slave-pflr krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986582, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:22.998217 7673 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:16:22.999099 7673 webserver.cc:460] Webserver started at http://127.24.20.129:42107/ using document root <none> and password file <none>
I20260620 20:16:22.999696 7673 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:22.999783 7673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:23.000089 7673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:23.002224 7673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "4c601a64c90040c1bf94eee2bf5e0d27"
format_stamp: "Formatted at 2026-06-20 20:16:22 on dist-test-slave-pflr"
server_key: "ec58f4d05e919d8b52e83df3172af91f"
server_key_iv: "32297125c1903f8ea4f6a02cfcf8fae8"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.002791 7673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "4c601a64c90040c1bf94eee2bf5e0d27"
format_stamp: "Formatted at 2026-06-20 20:16:22 on dist-test-slave-pflr"
server_key: "ec58f4d05e919d8b52e83df3172af91f"
server_key_iv: "32297125c1903f8ea4f6a02cfcf8fae8"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.006345 7673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:16:23.008715 7688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:23.009987 7673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:23.010136 7673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "4c601a64c90040c1bf94eee2bf5e0d27"
format_stamp: "Formatted at 2026-06-20 20:16:22 on dist-test-slave-pflr"
server_key: "ec58f4d05e919d8b52e83df3172af91f"
server_key_iv: "32297125c1903f8ea4f6a02cfcf8fae8"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.010250 7673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:23.030800 7673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:23.034018 7673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:23.034219 7673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:23.034816 7673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:23.035619 7673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:23.035689 7673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:23.035746 7673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:23.035800 7673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:23.044801 7673 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:39969
I20260620 20:16:23.044857 7801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:39969 every 8 connection(s)
I20260620 20:16:23.045665 7673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:23.050987 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 7673
I20260620 20:16:23.051115 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:16:23.051404 24658 external_mini_cluster.cc:1428] Setting key c672defa74bbb7a178c217d93d00d335
Jun 20 20:16:23 dist-test-slave-pflr krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986582, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:23.060083 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.047398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:59441 (local address 127.24.20.190:33277)
0620 20:16:23.047540 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.047545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.048367 (+ 822us) server_negotiation.cc:407] Connection header received
0620 20:16:23.049772 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.049775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.049834 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.049938 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:23.051282 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.051913 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.053397 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.053804 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.053936 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.056376 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:23.056379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:23.056382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:23.056403 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:23.058003 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:23.058590 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:23.058594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:23.058595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:23.058678 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:23.059020 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:23.059025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:23.059026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:23.059200 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:23.059305 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:23.059771 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 20:16:23.059912 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:16:23.060863 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.047800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:59441)
0620 20:16:23.048221 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:23.048261 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:23.049568 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:23.050106 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:23.050112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:23.050447 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:23.051048 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.051072 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.052042 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.052045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.053277 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.053287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.053973 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.053976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.054125 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.054761 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:23.054776 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:23.056208 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:23.058160 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:23.058165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:23.058178 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:23.058457 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:23.058774 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:23.058777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:23.058779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:23.058914 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:23.059303 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:23.059310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:23.059551 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 20:16:23.059731 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 20:16:23.060031 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:16:23.062233 7802 heartbeater.cc:347] Connected to a master server at 127.24.20.190:33277
I20260620 20:16:23.062541 7802 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:23.063009 7802 heartbeater.cc:511] Master 127.24.20.190:33277 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
I20260620 20:16:23.064694 7611 ts_manager.cc:194] Registered new tserver with Master: 4c601a64c90040c1bf94eee2bf5e0d27 (127.24.20.129:39969)
I20260620 20:16:23.066001 7611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:59441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:16:23.080152 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:33277
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:23.189419 7807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:23.189654 7807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:23.189697 7807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:23.193570 7807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:23.193619 7807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:23.193673 7807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:16:23.198791 7807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:23.199780 7807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:23.200850 7807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:23.209820 7815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:23.209834 7812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:23.209834 7813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:23.210072 7807 server_base.cc:1034] running on GCE node
I20260620 20:16:23.210485 7807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:23.211102 7807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:23.212261 7807 hybrid_clock.cc:648] HybridClock initialized: now 1781986583212247 us; error 26 us; skew 500 ppm
Jun 20 20:16:23 dist-test-slave-pflr krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986583, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:23.215114 7807 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:16:23.215878 7807 webserver.cc:460] Webserver started at http://127.24.20.130:43889/ using document root <none> and password file <none>
I20260620 20:16:23.216416 7807 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:23.216473 7807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:23.216691 7807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:23.218567 7807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "03b332032d224af28ccbdca76055772e"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-pflr"
server_key: "1b159e431479774825754c44ae24f130"
server_key_iv: "cf4ca4c09318dd893e5f83f686ea113e"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.219062 7807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "03b332032d224af28ccbdca76055772e"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-pflr"
server_key: "1b159e431479774825754c44ae24f130"
server_key_iv: "cf4ca4c09318dd893e5f83f686ea113e"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.222523 7807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:16:23.224722 7822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:23.225879 7807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:23.225975 7807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "03b332032d224af28ccbdca76055772e"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-pflr"
server_key: "1b159e431479774825754c44ae24f130"
server_key_iv: "cf4ca4c09318dd893e5f83f686ea113e"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.226100 7807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:23.272841 7807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:23.275714 7807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:23.275926 7807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:23.276511 7807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:23.277333 7807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:23.277405 7807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:23.277470 7807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:23.277514 7807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:23.286341 7807 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:36257
I20260620 20:16:23.286367 7937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:36257 every 8 connection(s)
I20260620 20:16:23.287227 7807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
Jun 20 20:16:23 dist-test-slave-pflr krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986583, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:23.297626 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 7807
I20260620 20:16:23.297791 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:16:23.298120 24658 external_mini_cluster.cc:1428] Setting key 313fb4693e535d620f5f666e840edb1a
I20260620 20:16:23.301949 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.288999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:41021 (local address 127.24.20.190:33277)
0620 20:16:23.289162 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.289167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.290055 (+ 888us) server_negotiation.cc:407] Connection header received
0620 20:16:23.291331 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.291334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.291381 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.291502 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:23.292664 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.293210 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.294283 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.294538 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.294622 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.297412 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:23.297416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:23.297419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:23.297450 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:23.299311 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:23.299975 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:23.299979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:23.299981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:23.300091 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:23.300529 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:23.300532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:23.300534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:23.300813 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:23.300898 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:23.301569 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 20:16:23.301719 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:16:23.303052 7940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.289468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:41021)
0620 20:16:23.289906 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:23.289949 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:23.291131 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:23.291662 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:23.291668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:23.292014 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:23.292472 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.292485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.293323 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.293326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.294175 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.294184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.294697 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.294700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.294846 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.295399 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:23.295421 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:23.296965 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:23.299456 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:23.299462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:23.299483 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:23.299866 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:23.300197 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:23.300203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:23.300208 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:23.300400 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:23.300916 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:23.300925 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:23.301255 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 20:16:23.301483 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 20:16:23.301853 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":119,"threads_started":1}
I20260620 20:16:23.304318 7938 heartbeater.cc:347] Connected to a master server at 127.24.20.190:33277
I20260620 20:16:23.304627 7938 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:23.305236 7938 heartbeater.cc:511] Master 127.24.20.190:33277 requested a full tablet report, sending...
I20260620 20:16:23.306396 7611 ts_manager.cc:194] Registered new tserver with Master: 03b332032d224af28ccbdca76055772e (127.24.20.130:36257)
I20260620 20:16:23.306983 7611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:41021
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:16:23.327575 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:33277
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:23.445521 7943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:23.445894 7943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:23.445968 7943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:23.451772 7943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:23.451857 7943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:23.451941 7943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:16:23.462162 7943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:23.463629 7943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:23.465094 7943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:23.476686 7948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:23.476737 7949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:23.476758 7951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:23.476907 7943 server_base.cc:1034] running on GCE node
I20260620 20:16:23.477253 7943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:23.477845 7943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:23.479007 7943 hybrid_clock.cc:648] HybridClock initialized: now 1781986583478988 us; error 29 us; skew 500 ppm
Jun 20 20:16:23 dist-test-slave-pflr krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986583, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:23.481762 7943 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:16:23.482512 7943 webserver.cc:460] Webserver started at http://127.24.20.131:46521/ using document root <none> and password file <none>
I20260620 20:16:23.482998 7943 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:23.483053 7943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:23.483207 7943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:16:23.485145 7943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "130e36847d074a8099c7306e54b194b4"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-pflr"
server_key: "dd89a9c0703dbf72c7e27a6e86b08d0a"
server_key_iv: "d730f9c9d103ec5947dc61a09f8a48bc"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.485634 7943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "130e36847d074a8099c7306e54b194b4"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-pflr"
server_key: "dd89a9c0703dbf72c7e27a6e86b08d0a"
server_key_iv: "d730f9c9d103ec5947dc61a09f8a48bc"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.489078 7943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:16:23.491384 7958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:23.492871 7943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:16:23.492961 7943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "130e36847d074a8099c7306e54b194b4"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-pflr"
server_key: "dd89a9c0703dbf72c7e27a6e86b08d0a"
server_key_iv: "d730f9c9d103ec5947dc61a09f8a48bc"
server_key_version: "encryptionkey@0"
I20260620 20:16:23.493085 7943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:23.506984 7943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:23.510032 7943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:23.510223 7943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:23.510782 7943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:23.511574 7943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:23.511618 7943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:23.511685 7943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:23.511718 7943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:23.521143 7943 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:43383
I20260620 20:16:23.521167 8072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:43383 every 8 connection(s)
I20260620 20:16:23.522056 7943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:23.523609 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 7943
I20260620 20:16:23.523731 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:16:23.524019 24658 external_mini_cluster.cc:1428] Setting key f7a383ea5a179558edc85044ac9aa720
Jun 20 20:16:23 dist-test-slave-pflr krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986583, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:23.535872 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.523853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:45297 (local address 127.24.20.190:33277)
0620 20:16:23.523995 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.524004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.524957 (+ 953us) server_negotiation.cc:407] Connection header received
0620 20:16:23.526511 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.526513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.526558 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.526647 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:23.527811 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.528337 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.529666 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.529920 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.530018 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.532321 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:23.532324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:23.532326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:23.532350 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:23.533630 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:23.534166 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:23.534169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:23.534170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:23.534242 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:23.534576 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:23.534578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:23.534579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:23.534738 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:23.534860 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:23.535591 (+ 731us) server_negotiation.cc:299] Negotiation successful
0620 20:16:23.535707 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:16:23.536623 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.524294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:45297)
0620 20:16:23.524786 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:23.524837 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:23.526286 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:23.526832 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:23.526842 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:23.527210 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:23.527669 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.527683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.528479 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.528482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.529558 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.529568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.530056 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.530059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.530225 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.530805 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:23.530820 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:23.532181 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:23.533746 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:23.533751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:23.533761 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:23.534081 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:23.534329 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:23.534331 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:23.534333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:23.534495 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:23.534873 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:23.534882 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:23.535190 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 20:16:23.535524 (+ 334us) client_negotiation.cc:240] Negotiation successful
0620 20:16:23.535822 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":133,"threads_started":1}
I20260620 20:16:23.537698 8073 heartbeater.cc:347] Connected to a master server at 127.24.20.190:33277
I20260620 20:16:23.537955 8073 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:23.538416 8073 heartbeater.cc:511] Master 127.24.20.190:33277 requested a full tablet report, sending...
I20260620 20:16:23.539373 7611 ts_manager.cc:194] Registered new tserver with Master: 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383)
I20260620 20:16:23.539923 7611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:45297
I20260620 20:16:23.550328 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:16:23.559894 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.552304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.24.20.190:33277)
0620 20:16:23.552429 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.552432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.552777 (+ 345us) server_negotiation.cc:407] Connection header received
0620 20:16:23.552927 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.552929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.552969 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.553041 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:23.553758 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.554264 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.555146 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.555349 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.555429 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.556845 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:23.556848 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:23.556850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:23.556872 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:23.558258 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:23.558637 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:23.558640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:23.558641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:23.558720 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:23.558956 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:23.558959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:23.558960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:23.559174 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:23.559285 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:23.559658 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 20:16:23.559770 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:16:23.562799 7611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:16:23.566069 7611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:16:23.580318 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.574682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:43383 (local address 127.0.0.1:35138)
0620 20:16:23.575592 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:23.575644 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:23.575843 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:23.576631 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:23.576634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:23.576650 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:23.576762 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.576769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.577760 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.577766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.578975 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.578988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.579766 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.579770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.579904 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.579950 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 20:16:23.580063 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 20:16:23.580173 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":5,"spinlock_wait_cycles":7168,"thread_start_us":96,"threads_started":1}
I20260620 20:16:23.580351 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.574844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35138 (local address 127.24.20.131:43383)
0620 20:16:23.575445 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.575457 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.575730 (+ 273us) server_negotiation.cc:407] Connection header received
0620 20:16:23.576032 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.576038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.576280 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.576460 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:23.576882 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.577632 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.579101 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.579637 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.579767 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.579919 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:23.580096 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 20:16:23.580204 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":144,"threads_started":1}
I20260620 20:16:23.581395 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.574513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:36257 (local address 127.0.0.1:51378)
0620 20:16:23.575592 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:23.575644 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:23.575843 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:23.576444 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:23.576449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:23.576470 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:23.576629 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.576635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.578240 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.578247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.579873 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.579881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.580993 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.580996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.581095 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.581117 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:16:23.581185 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 20:16:23.581261 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":131,"spinlock_wait_cycles":3200,"thread_start_us":74,"threads_started":1}
I20260620 20:16:23.582011 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.574660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.24.20.130:36257)
0620 20:16:23.575307 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.575321 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.575730 (+ 409us) server_negotiation.cc:407] Connection header received
0620 20:16:23.576059 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.576065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.576274 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.576444 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:23.576767 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.578072 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.579999 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.580857 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.580992 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.581794 (+ 802us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:23.581834 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 20:16:23.581900 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":109,"threads_started":1}
I20260620 20:16:23.582515 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.575617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:39969 (local address 127.0.0.1:59158)
0620 20:16:23.576075 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:23.576086 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:23.576157 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:23.576792 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:23.576798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:23.576819 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:23.576892 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.576897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.578308 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.578311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.579869 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.579881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.581261 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.581265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.582211 (+ 946us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.582242 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 20:16:23.582322 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 20:16:23.582398 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":165,"thread_start_us":93,"threads_started":1}
I20260620 20:16:23.582818 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.575924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.24.20.129:39969)
0620 20:16:23.576322 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.576330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.576355 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:16:23.576431 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.576436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.576642 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.576827 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:23.576975 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.578183 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.580027 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.580952 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.581060 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.581192 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:23.582391 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0620 20:16:23.582497 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":119,"threads_started":1}
I20260620 20:16:23.583720 8007 tablet_service.cc:1467] Processing CreateTablet for tablet 8117b97a8dba438ea9fa9b96275e0c7c (DEFAULT_TABLE table=test-table [id=20eb100b4acb47348432bb577dda05c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:23.584779 8007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8117b97a8dba438ea9fa9b96275e0c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:23.584573 7871 tablet_service.cc:1467] Processing CreateTablet for tablet 8117b97a8dba438ea9fa9b96275e0c7c (DEFAULT_TABLE table=test-table [id=20eb100b4acb47348432bb577dda05c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:23.585592 7871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8117b97a8dba438ea9fa9b96275e0c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:23.585423 7736 tablet_service.cc:1467] Processing CreateTablet for tablet 8117b97a8dba438ea9fa9b96275e0c7c (DEFAULT_TABLE table=test-table [id=20eb100b4acb47348432bb577dda05c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:23.586334 7736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8117b97a8dba438ea9fa9b96275e0c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:23.591329 8090 tablet_bootstrap.cc:492] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4: Bootstrap starting.
I20260620 20:16:23.592394 8091 tablet_bootstrap.cc:492] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e: Bootstrap starting.
I20260620 20:16:23.593431 8092 tablet_bootstrap.cc:492] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Bootstrap starting.
I20260620 20:16:23.593992 8090 tablet_bootstrap.cc:654] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:23.595146 8092 tablet_bootstrap.cc:654] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:23.595182 8091 tablet_bootstrap.cc:654] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:23.595754 8090 log.cc:824] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:23.596000 8092 log.cc:824] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:23.596094 8091 log.cc:824] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:23.597941 8091 tablet_bootstrap.cc:492] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e: No bootstrap required, opened a new log
I20260620 20:16:23.598045 8092 tablet_bootstrap.cc:492] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: No bootstrap required, opened a new log
I20260620 20:16:23.598044 8090 tablet_bootstrap.cc:492] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4: No bootstrap required, opened a new log
I20260620 20:16:23.598162 8091 ts_tablet_manager.cc:1397] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:16:23.598237 8092 ts_tablet_manager.cc:1397] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:16:23.598237 8090 ts_tablet_manager.cc:1397] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 20:16:23.601146 8091 raft_consensus.cc:348] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:23.601382 8091 raft_consensus.cc:374] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:23.601451 8091 raft_consensus.cc:729] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b332032d224af28ccbdca76055772e, State: Initialized, Role: FOLLOWER
I20260620 20:16:23.601898 8091 consensus_queue.cc:260] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:23.602732 8090 raft_consensus.cc:348] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:23.602957 8091 ts_tablet_manager.cc:1428] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:16:23.603020 8090 raft_consensus.cc:374] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:23.603086 8090 raft_consensus.cc:729] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130e36847d074a8099c7306e54b194b4, State: Initialized, Role: FOLLOWER
I20260620 20:16:23.603312 7938 heartbeater.cc:503] Master 127.24.20.190:33277 was elected leader, sending a full tablet report...
I20260620 20:16:23.603200 8092 raft_consensus.cc:348] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:23.603473 8092 raft_consensus.cc:374] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:23.603526 8092 raft_consensus.cc:729] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c601a64c90040c1bf94eee2bf5e0d27, State: Initialized, Role: FOLLOWER
I20260620 20:16:23.603576 8090 consensus_queue.cc:260] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:23.603930 8092 consensus_queue.cc:260] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:23.604209 8073 heartbeater.cc:503] Master 127.24.20.190:33277 was elected leader, sending a full tablet report...
I20260620 20:16:23.604916 8092 ts_tablet_manager.cc:1428] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 20:16:23.605118 8090 ts_tablet_manager.cc:1428] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 20:16:23.605285 7802 heartbeater.cc:503] Master 127.24.20.190:33277 was elected leader, sending a full tablet report...
I20260620 20:16:23.752722 8098 raft_consensus.cc:482] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:23.752962 8098 raft_consensus.cc:504] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:23.754136 8098 leader_election.cc:288] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03b332032d224af28ccbdca76055772e (127.24.20.130:36257), 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383)
I20260620 20:16:23.758333 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.754560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:33909 (local address 127.24.20.131:43383)
0620 20:16:23.754696 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.754701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.754906 (+ 205us) server_negotiation.cc:407] Connection header received
0620 20:16:23.755081 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.755084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.755132 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.755216 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:23.755519 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.756029 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.757281 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.757672 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.757798 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.757848 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:23.758107 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 20:16:23.758206 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:16:23.758342 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.754416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:43383 (local address 127.24.20.129:33909)
0620 20:16:23.754780 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:23.754796 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:23.754958 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:23.755228 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:23.755231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:23.755246 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:23.755393 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.755399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.756151 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.756156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.757155 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.757165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.757798 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.757801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.757910 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.757943 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 20:16:23.758053 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 20:16:23.758136 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":55,"thread_start_us":129,"threads_started":1}
I20260620 20:16:23.758478 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.754590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:60899 (local address 127.24.20.130:36257)
0620 20:16:23.754780 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.754786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.754937 (+ 151us) server_negotiation.cc:407] Connection header received
0620 20:16:23.755115 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.755118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.755167 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.755250 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:23.755582 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.756057 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.757279 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.757668 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.757798 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.757848 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:23.758277 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 20:16:23.758358 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 20:16:23.758900 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.754503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:36257 (local address 127.24.20.129:60899)
0620 20:16:23.754840 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:23.754851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:23.755003 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:23.755259 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:23.755263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:23.755299 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:23.755457 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.755462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.756170 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.756173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.757147 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:23.757156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.757910 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.757914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:23.758087 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.758112 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:16:23.758238 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 20:16:23.758303 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":58,"spinlock_wait_cycles":2944,"thread_start_us":100,"threads_started":1}
I20260620 20:16:23.759054 8027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8117b97a8dba438ea9fa9b96275e0c7c" candidate_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130e36847d074a8099c7306e54b194b4" is_pre_election: true
I20260620 20:16:23.759426 8027 raft_consensus.cc:2454] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c601a64c90040c1bf94eee2bf5e0d27 in term 0.
I20260620 20:16:23.759567 7892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8117b97a8dba438ea9fa9b96275e0c7c" candidate_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03b332032d224af28ccbdca76055772e" is_pre_election: true
I20260620 20:16:23.759863 7892 raft_consensus.cc:2454] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c601a64c90040c1bf94eee2bf5e0d27 in term 0.
I20260620 20:16:23.759927 7690 leader_election.cc:302] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 130e36847d074a8099c7306e54b194b4, 4c601a64c90040c1bf94eee2bf5e0d27; no voters:
I20260620 20:16:23.760221 8098 raft_consensus.cc:2781] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:16:23.760339 8098 raft_consensus.cc:482] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:16:23.760399 8098 raft_consensus.cc:3037] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:23.761425 8098 raft_consensus.cc:504] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:23.761757 8098 leader_election.cc:288] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 1 election: Requested vote from peers 03b332032d224af28ccbdca76055772e (127.24.20.130:36257), 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383)
I20260620 20:16:23.762126 8027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8117b97a8dba438ea9fa9b96275e0c7c" candidate_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130e36847d074a8099c7306e54b194b4"
I20260620 20:16:23.762115 7892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8117b97a8dba438ea9fa9b96275e0c7c" candidate_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03b332032d224af28ccbdca76055772e"
I20260620 20:16:23.762236 7892 raft_consensus.cc:3037] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:23.762236 8027 raft_consensus.cc:3037] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:23.763497 7892 raft_consensus.cc:2454] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c601a64c90040c1bf94eee2bf5e0d27 in term 1.
I20260620 20:16:23.763497 8027 raft_consensus.cc:2454] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c601a64c90040c1bf94eee2bf5e0d27 in term 1.
I20260620 20:16:23.763792 7692 leader_election.cc:302] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03b332032d224af28ccbdca76055772e, 130e36847d074a8099c7306e54b194b4, 4c601a64c90040c1bf94eee2bf5e0d27; no voters:
I20260620 20:16:23.763942 8098 raft_consensus.cc:2781] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:23.764159 8098 raft_consensus.cc:686] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 LEADER]: Becoming Leader. State: Replica: 4c601a64c90040c1bf94eee2bf5e0d27, State: Running, Role: LEADER
I20260620 20:16:23.764492 8098 consensus_queue.cc:237] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:23.767396 7610 catalog_manager.cc:5495] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c601a64c90040c1bf94eee2bf5e0d27 (127.24.20.129). New cstate: current_term: 1 leader_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } health_report { overall_health: HEALTHY } } }
W20260620 20:16:23.772902 8074 tablet.cc:2369] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:16:23.788254 7939 tablet.cc:2369] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:16:23.797026 7803 tablet.cc:2369] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:16:23.813271 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.809691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59166 (local address 127.24.20.129:39969)
0620 20:16:23.809822 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:23.809855 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:23.809872 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:16:23.809988 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:23.809991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:23.810045 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:23.810157 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:23.810412 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.810944 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.811790 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:23.812024 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:23.812134 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:23.812433 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:23.812874 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:23.813017 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:23.813073 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 20:16:23.813127 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:16:23.818970 8027 raft_consensus.cc:1261] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4 [term 1 FOLLOWER]: Refusing update from remote peer 4c601a64c90040c1bf94eee2bf5e0d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:23.818976 7892 raft_consensus.cc:1261] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e [term 1 FOLLOWER]: Refusing update from remote peer 4c601a64c90040c1bf94eee2bf5e0d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:23.819679 8101 consensus_queue.cc:1035] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:23.819849 8098 consensus_queue.cc:1035] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:23.828123 8106 mvcc.cc:205] Tried to move back new op lower bound from 7299017047315238912 to 7299017047101808640. Current Snapshot: MvccSnapshot[applied={T|T < 7299017047315238912}]
I20260620 20:16:23.828297 8105 mvcc.cc:205] Tried to move back new op lower bound from 7299017047315238912 to 7299017047101808640. Current Snapshot: MvccSnapshot[applied={T|T < 7299017047315238912}]
I20260620 20:16:23.834396 7611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:23.834578 7611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:23.836798 7611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Sending DeleteTablet for 3 replicas of tablet 8117b97a8dba438ea9fa9b96275e0c7c
I20260620 20:16:23.837603 7871 tablet_service.cc:1514] Processing DeleteTablet for tablet 8117b97a8dba438ea9fa9b96275e0c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:23 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:51378
I20260620 20:16:23.837677 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 7673
I20260620 20:16:23.837656 8007 tablet_service.cc:1514] Processing DeleteTablet for tablet 8117b97a8dba438ea9fa9b96275e0c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:23 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:35138
I20260620 20:16:23.838423 8115 tablet_replica.cc:330] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4: stopping tablet replica
I20260620 20:16:23.838687 8115 raft_consensus.cc:2229] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:23.838972 8115 raft_consensus.cc:2258] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:23.839855 8114 tablet_replica.cc:330] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e: stopping tablet replica
I20260620 20:16:23.840137 8114 raft_consensus.cc:2229] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:23.840261 8115 ts_tablet_manager.cc:1905] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:23.840466 8114 raft_consensus.cc:2258] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:23.841595 8114 ts_tablet_manager.cc:1905] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:23.842849 8115 ts_tablet_manager.cc:1918] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:23.842948 8115 log.cc:1196] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/8117b97a8dba438ea9fa9b96275e0c7c
I20260620 20:16:23.843263 8115 ts_tablet_manager.cc:1939] T 8117b97a8dba438ea9fa9b96275e0c7c P 130e36847d074a8099c7306e54b194b4: Deleting consensus metadata
I20260620 20:16:23.843763 8114 ts_tablet_manager.cc:1918] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:23.843852 8114 log.cc:1196] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/8117b97a8dba438ea9fa9b96275e0c7c
W20260620 20:16:23.844137 7595 connection.cc:535] client connection to 127.24.20.129:39969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:16:23.844182 8114 ts_tablet_manager.cc:1939] T 8117b97a8dba438ea9fa9b96275e0c7c P 03b332032d224af28ccbdca76055772e: Deleting consensus metadata
W20260620 20:16:23.844271 7595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:16:23.844573 7596 catalog_manager.cc:4841] TS 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383): tablet 8117b97a8dba438ea9fa9b96275e0c7c (table test-table [id=20eb100b4acb47348432bb577dda05c8]) successfully deleted
I20260620 20:16:23.845016 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 7807
I20260620 20:16:23.845753 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.845260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:39969 (local address 127.0.0.1:59182)
0620 20:16:23.845636 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:23.845689 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:39969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305}
W20260620 20:16:23.845927 7595 catalog_manager.cc:4584] TS 4c601a64c90040c1bf94eee2bf5e0d27 (127.24.20.129:39969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8117b97a8dba438ea9fa9b96275e0c7c: Network error: Client connection negotiation failed: client connection to 127.24.20.129:39969: connect: Connection refused (error 111)
I20260620 20:16:23.850977 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 7943
I20260620 20:16:23.851828 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:23.851128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:36257 (local address 127.0.0.1:51382)
0620 20:16:23.851670 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:23.851734 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.130:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454}
I20260620 20:16:23.856897 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 7579
I20260620 20:16:43.863225 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:33277
--webserver_interface=127.24.20.190
--webserver_port=34581
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:33277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:16:43.971171 8117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:43.971408 8117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:43.971446 8117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:43.975359 8117 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:16:43.975406 8117 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:16:43.975422 8117 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:16:43.975436 8117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:16:43.975492 8117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:43.975526 8117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:43.975550 8117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:16:43.975605 8117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:16:43.980743 8117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:33277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:33277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=34581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:43.981695 8117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:43.982690 8117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:43.995199 8126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:43.995240 8123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:43.995361 8117 server_base.cc:1034] running on GCE node
W20260620 20:16:43.995199 8124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:43.996132 8117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:43.997287 8117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:43.998457 8117 hybrid_clock.cc:648] HybridClock initialized: now 1781986603998437 us; error 31 us; skew 500 ppm
Jun 20 20:16:43 dist-test-slave-pflr krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986603, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:44.001119 8117 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:16:44.002030 8117 webserver.cc:460] Webserver started at http://127.24.20.190:34581/ using document root <none> and password file <none>
I20260620 20:16:44.002607 8117 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:44.002709 8117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:44.006575 8117 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 20:16:44.008790 8133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:44.010222 8117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:44.010504 8117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "e88230dbd74442f6a7f9e53b69001873"
format_stamp: "Formatted at 2026-06-20 20:16:22 on dist-test-slave-pflr"
server_key: "9dd2dd2aae897c3cb90c226b3c700175"
server_key_iv: "7d4a0ab0f0ad299cab4e3c809569753e"
server_key_version: "encryptionkey@0"
I20260620 20:16:44.011104 8117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:44.033399 8117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:44.036231 8117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:44.036441 8117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:44.043720 8117 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:33277
I20260620 20:16:44.044008 8185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:33277 every 8 connection(s)
I20260620 20:16:44.044870 8117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:16:44.049201 8186 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:16:44.049634 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 8117
I20260620 20:16:44.050081 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:39969
--local_ip_for_outbound_sockets=127.24.20.129
--tserver_master_addrs=127.24.20.190:33277
--webserver_port=42107
--webserver_interface=127.24.20.129
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:16:44.050935 8186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Bootstrap starting.
I20260620 20:16:44.067306 8186 log.cc:824] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:44.075526 8186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:16:44.075897 8186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Bootstrap complete.
I20260620 20:16:44.080008 8186 raft_consensus.cc:348] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } }
I20260620 20:16:44.080403 8186 raft_consensus.cc:729] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e88230dbd74442f6a7f9e53b69001873, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.080974 8186 consensus_queue.cc:260] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } }
I20260620 20:16:44.081118 8186 raft_consensus.cc:388] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:16:44.081213 8186 raft_consensus.cc:482] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:16:44.081312 8186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:16:44.083149 8186 raft_consensus.cc:504] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } }
I20260620 20:16:44.083487 8186 leader_election.cc:302] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e88230dbd74442f6a7f9e53b69001873; no voters:
I20260620 20:16:44.083817 8186 leader_election.cc:288] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:16:44.083935 8190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:16:44.084249 8190 raft_consensus.cc:686] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 2 LEADER]: Becoming Leader. State: Replica: e88230dbd74442f6a7f9e53b69001873, State: Running, Role: LEADER
I20260620 20:16:44.084558 8190 consensus_queue.cc:237] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } }
I20260620 20:16:44.085071 8186 sys_catalog.cc:564] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:16:44.085515 8192 sys_catalog.cc:455] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e88230dbd74442f6a7f9e53b69001873. Latest consensus state: current_term: 2 leader_uuid: "e88230dbd74442f6a7f9e53b69001873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } } }
I20260620 20:16:44.085633 8192 sys_catalog.cc:458] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:44.086217 8191 sys_catalog.cc:455] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e88230dbd74442f6a7f9e53b69001873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } } }
I20260620 20:16:44.086301 8191 sys_catalog.cc:458] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: This master's current role is: LEADER
I20260620 20:16:44.086671 8194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:16:44.091256 8194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:16:44.091873 8194 catalog_manager.cc:1259] Loaded cluster ID: eed7674130714db094826ae2c66bbd30
I20260620 20:16:44.091928 8194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:16:44.094440 8194 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:16:44.095291 8194 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Loaded TSK: 0
I20260620 20:16:44.109272 8194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Generated new TSK 1
I20260620 20:16:44.109525 8194 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:16:44.166194 8188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:44.166432 8188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:44.166473 8188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:44.170501 8188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:44.170557 8188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:44.170615 8188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:16:44.175930 8188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:39969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=42107
--tserver_master_addrs=127.24.20.190:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:44.176998 8188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:44.178009 8188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:44.187700 8216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:44.187690 8213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:44.187707 8214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:44.188014 8188 server_base.cc:1034] running on GCE node
I20260620 20:16:44.188423 8188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:44.189123 8188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:44.190306 8188 hybrid_clock.cc:648] HybridClock initialized: now 1781986604190278 us; error 45 us; skew 500 ppm
Jun 20 20:16:44 dist-test-slave-pflr krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986604, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:44.193269 8188 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:16:44.194321 8188 webserver.cc:460] Webserver started at http://127.24.20.129:42107/ using document root <none> and password file <none>
I20260620 20:16:44.195017 8188 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:44.195125 8188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:44.200151 8188 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:44.202858 8223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:44.204407 8188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:44.204524 8188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "4c601a64c90040c1bf94eee2bf5e0d27"
format_stamp: "Formatted at 2026-06-20 20:16:22 on dist-test-slave-pflr"
server_key: "ec58f4d05e919d8b52e83df3172af91f"
server_key_iv: "32297125c1903f8ea4f6a02cfcf8fae8"
server_key_version: "encryptionkey@0"
I20260620 20:16:44.205241 8188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:44.232157 8188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:44.235503 8188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:44.235702 8188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:44.236310 8188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:44.237361 8230 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 20:16:44.241394 8188 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 20:16:44.241463 8188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 20:16:44.241511 8188 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 20:16:44.243116 8188 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 20:16:44.243158 8188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 20:16:44.243444 8230 tablet_bootstrap.cc:492] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Bootstrap starting.
I20260620 20:16:44.253489 8188 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:39969
I20260620 20:16:44.253564 8338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:39969 every 8 connection(s)
I20260620 20:16:44.254393 8188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:16:44.257001 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 8188
I20260620 20:16:44.257510 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:36257
--local_ip_for_outbound_sockets=127.24.20.130
--tserver_master_addrs=127.24.20.190:33277
--webserver_port=43889
--webserver_interface=127.24.20.130
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:16:44.263322 8230 log.cc:824] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:16:44 dist-test-slave-pflr krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986604, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:44.271548 8230 tablet_bootstrap.cc:492] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:16:44.271914 8230 tablet_bootstrap.cc:492] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Bootstrap complete.
I20260620 20:16:44.273214 8230 ts_tablet_manager.cc:1397] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.004s
I20260620 20:16:44.274084 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.256402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:36411 (local address 127.24.20.190:33277)
0620 20:16:44.256788 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.256809 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.257357 (+ 548us) server_negotiation.cc:407] Connection header received
0620 20:16:44.259020 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.259036 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.259491 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.259941 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:44.261692 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.262788 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.264131 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.265002 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.265198 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.268306 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:44.268312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:44.268327 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:44.268356 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:44.270530 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.271126 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.271132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.271137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.271279 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.271692 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.271695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.271697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.272115 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:44.272218 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.272936 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.273229 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":135,"threads_started":1}
I20260620 20:16:44.274091 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.256384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:36411)
0620 20:16:44.257172 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.257227 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.258515 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.260425 (+ 1910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.260434 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.260919 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:44.261514 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.261532 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.262948 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.262953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.263983 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.263994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.265155 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.265159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.265336 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.266073 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:44.266093 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:44.268090 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:44.270657 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:44.270663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:44.270677 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:44.271005 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:44.271397 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:44.271400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:44.271403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:44.271574 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:44.272299 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:44.272305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:44.272581 (+ 276us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.272797 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.273103 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":162,"threads_started":1}
I20260620 20:16:44.275928 8230 raft_consensus.cc:348] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:44.276429 8230 raft_consensus.cc:729] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c601a64c90040c1bf94eee2bf5e0d27, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.276945 8230 consensus_queue.cc:260] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:44.277076 8339 heartbeater.cc:347] Connected to a master server at 127.24.20.190:33277
I20260620 20:16:44.277297 8339 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:44.277806 8339 heartbeater.cc:511] Master 127.24.20.190:33277 requested a full tablet report, sending...
I20260620 20:16:44.277908 8230 ts_tablet_manager.cc:1428] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:16:44.280113 8150 ts_manager.cc:194] Registered new tserver with Master: 4c601a64c90040c1bf94eee2bf5e0d27 (127.24.20.129:39969)
I20260620 20:16:44.283437 8150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:36411
I20260620 20:16:44.285835 8339 heartbeater.cc:503] Master 127.24.20.190:33277 was elected leader, sending a full tablet report...
Jun 20 20:16:44 dist-test-slave-pflr krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986603, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.129@KRBTEST.COM
I20260620 20:16:44.294026 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.282983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.24.20.129:39969)
0620 20:16:44.283709 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.283719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.283740 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:16:44.283802 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.283807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.284008 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.284132 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:44.284994 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.286095 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.287281 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.287602 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.287734 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.290130 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:44.290139 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:44.290151 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:44.290176 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:44.292017 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.292484 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.292491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.292493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.292584 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.292932 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.292934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.292936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.293186 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:44.293293 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.293647 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.293905 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":84,"threads_started":1}
I20260620 20:16:44.294030 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.282913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:39969 (local address 127.0.0.1:60758)
0620 20:16:44.283294 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.283333 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.283456 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.284285 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.284291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.284726 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:44.284864 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.284872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.286225 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.286231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.287157 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.287165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.287733 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.287736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.287842 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.288441 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:44.288457 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:44.289994 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:44.292131 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:44.292139 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:44.292143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:44.292369 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:44.292722 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:44.292726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:44.292728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:44.292821 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:44.293300 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:44.293306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:44.293433 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.293578 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.293858 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":123,"threads_started":1}
I20260620 20:16:44.295074 8273 tablet_service.cc:1514] Processing DeleteTablet for tablet 8117b97a8dba438ea9fa9b96275e0c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:23 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:60758
I20260620 20:16:44.295866 8348 tablet_replica.cc:330] stopping tablet replica
I20260620 20:16:44.296097 8348 raft_consensus.cc:2229] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:44.296283 8348 raft_consensus.cc:2258] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:44.297173 8348 ts_tablet_manager.cc:1905] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:44.300403 8348 ts_tablet_manager.cc:1918] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:44.300478 8348 log.cc:1196] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal/wals/8117b97a8dba438ea9fa9b96275e0c7c
I20260620 20:16:44.300778 8348 ts_tablet_manager.cc:1939] T 8117b97a8dba438ea9fa9b96275e0c7c P 4c601a64c90040c1bf94eee2bf5e0d27: Deleting consensus metadata
I20260620 20:16:44.302357 8134 catalog_manager.cc:4841] TS 4c601a64c90040c1bf94eee2bf5e0d27 (127.24.20.129:39969): tablet 8117b97a8dba438ea9fa9b96275e0c7c (table test-table [id=20eb100b4acb47348432bb577dda05c8]) successfully deleted
W20260620 20:16:44.375584 8343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:44.375836 8343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:44.375885 8343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:44.380314 8343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:44.380383 8343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:44.380442 8343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:16:44.386252 8343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:36257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=43889
--tserver_master_addrs=127.24.20.190:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:44.387270 8343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:44.388180 8343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:44.397513 8353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:44.397594 8354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:44.397604 8356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:44.397891 8343 server_base.cc:1034] running on GCE node
I20260620 20:16:44.398274 8343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:44.398887 8343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:44.400071 8343 hybrid_clock.cc:648] HybridClock initialized: now 1781986604400058 us; error 30 us; skew 500 ppm
Jun 20 20:16:44 dist-test-slave-pflr krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986604, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:44.402849 8343 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:16:44.403631 8343 webserver.cc:460] Webserver started at http://127.24.20.130:43889/ using document root <none> and password file <none>
I20260620 20:16:44.404227 8343 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:44.404330 8343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:44.407979 8343 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 20:16:44.410000 8363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:44.411255 8343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:16:44.411372 8343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "03b332032d224af28ccbdca76055772e"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-pflr"
server_key: "1b159e431479774825754c44ae24f130"
server_key_iv: "cf4ca4c09318dd893e5f83f686ea113e"
server_key_version: "encryptionkey@0"
I20260620 20:16:44.411929 8343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:44.439062 8343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:44.442219 8343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:44.442422 8343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:44.442987 8343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:44.443758 8343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:44.443825 8343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:44.443892 8343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:44.443941 8343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:44.452286 8343 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:36257
I20260620 20:16:44.452342 8477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:36257 every 8 connection(s)
I20260620 20:16:44.453310 8343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:16:44.454890 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 8343
I20260620 20:16:44.455384 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:43383
--local_ip_for_outbound_sockets=127.24.20.131
--tserver_master_addrs=127.24.20.190:33277
--webserver_port=46521
--webserver_interface=127.24.20.131
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:16:44 dist-test-slave-pflr krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986604, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:44.467581 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.455067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:36363 (local address 127.24.20.190:33277)
0620 20:16:44.455202 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.455205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.456778 (+ 1573us) server_negotiation.cc:407] Connection header received
0620 20:16:44.458159 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.458162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.458211 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.458287 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:44.459475 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.460007 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.461223 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.461463 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.461583 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.463992 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:44.463996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:44.463998 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:44.464021 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:44.465534 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.466046 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.466050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.466052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.466133 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.466475 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.466479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.466480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.466665 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:44.466774 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.467257 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.467391 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:16:44.468321 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.455547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:36363)
0620 20:16:44.456575 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.456668 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.457885 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.458486 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.458494 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.458840 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:44.459299 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.459312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.460140 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.460144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.461094 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.461104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.461626 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.461630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.461799 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.462436 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:44.462452 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:44.463819 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:44.465664 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:44.465670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:44.465683 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:44.465942 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:44.466231 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:44.466235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:44.466237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:44.466373 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:44.466782 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:44.466792 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:44.467025 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.467214 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.467519 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":706,"threads_started":1}
I20260620 20:16:44.469471 8478 heartbeater.cc:347] Connected to a master server at 127.24.20.190:33277
I20260620 20:16:44.469731 8478 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:44.470206 8478 heartbeater.cc:511] Master 127.24.20.190:33277 requested a full tablet report, sending...
I20260620 20:16:44.471241 8150 ts_manager.cc:194] Registered new tserver with Master: 03b332032d224af28ccbdca76055772e (127.24.20.130:36257)
I20260620 20:16:44.471963 8150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:36363
W20260620 20:16:44.564558 8481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:16:44.564805 8481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:16:44.564844 8481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:16:44.568769 8481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:16:44.568816 8481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:16:44.568871 8481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:16:44.573832 8481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:43383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=46521
--tserver_master_addrs=127.24.20.190:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:16:44.574709 8481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:16:44.575598 8481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:16:44.584609 8486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:44.584676 8481 server_base.cc:1034] running on GCE node
W20260620 20:16:44.584576 8489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:16:44.584576 8487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:16:44.585248 8481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:16:44.585857 8481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:16:44.587132 8481 hybrid_clock.cc:648] HybridClock initialized: now 1781986604587077 us; error 58 us; skew 500 ppm
Jun 20 20:16:44 dist-test-slave-pflr krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986604, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:16:44.589880 8481 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:16:44.590642 8481 webserver.cc:460] Webserver started at http://127.24.20.131:46521/ using document root <none> and password file <none>
I20260620 20:16:44.591151 8481 fs_manager.cc:359] Metadata directory not provided
I20260620 20:16:44.591221 8481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:16:44.594920 8481 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:16:44.596858 8496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:44.598016 8481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:16:44.598157 8481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "130e36847d074a8099c7306e54b194b4"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-pflr"
server_key: "dd89a9c0703dbf72c7e27a6e86b08d0a"
server_key_iv: "d730f9c9d103ec5947dc61a09f8a48bc"
server_key_version: "encryptionkey@0"
I20260620 20:16:44.598664 8481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:16:44.623078 8481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:16:44.625742 8481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:16:44.625918 8481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:16:44.626446 8481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:16:44.627210 8481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:16:44.627279 8481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:44.627345 8481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:16:44.627395 8481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:16:44.635852 8481 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:43383
I20260620 20:16:44.635881 8610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:43383 every 8 connection(s)
I20260620 20:16:44.636875 8481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:16:44.642268 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 8481
Jun 20 20:16:44 dist-test-slave-pflr krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986604, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:16:44.651243 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.642916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50804 (local address 127.24.20.190:33277)
0620 20:16:44.643183 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.643186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.643311 (+ 125us) server_negotiation.cc:407] Connection header received
0620 20:16:44.643467 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.643469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.643506 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.643592 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:44.644458 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.645031 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.646158 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.646368 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.646439 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.648005 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:44.648007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:44.648009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:44.648041 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:44.649597 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.650001 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.650008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.650013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.650100 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.650400 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.650404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.650405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.650589 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:44.650795 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.650930 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.651053 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"spinlock_wait_cycles":4864,"thread_start_us":95,"threads_started":1}
I20260620 20:16:44.651599 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.638568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:38759 (local address 127.24.20.190:33277)
0620 20:16:44.638710 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.638716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.639545 (+ 829us) server_negotiation.cc:407] Connection header received
0620 20:16:44.640875 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.640878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.640926 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.641001 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:44.642199 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.642902 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.644644 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.644871 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.644982 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.647668 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:44.647671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:44.647673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:44.647692 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:44.649534 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.650055 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.650060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.650062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.650142 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.650487 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.650492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.650494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.650659 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:44.650764 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.651364 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.651463 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:16:44.652537 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.638974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:38759)
0620 20:16:44.639402 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.639441 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.640691 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.641171 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.641177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.641531 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:16:44.642016 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.642041 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.643044 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.643049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.644498 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.644507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.645020 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.645025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.645224 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.645795 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:16:44.645814 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:16:44.647497 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:16:44.649668 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:44.649674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:44.649686 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:44.649949 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:44.650240 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:16:44.650243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:16:44.650245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:16:44.650389 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:16:44.650774 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:16:44.650783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:16:44.651100 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.651306 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.651638 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260620 20:16:44.653644 8611 heartbeater.cc:347] Connected to a master server at 127.24.20.190:33277
I20260620 20:16:44.653832 8611 heartbeater.cc:464] Registering TS with master...
I20260620 20:16:44.654234 8611 heartbeater.cc:511] Master 127.24.20.190:33277 requested a full tablet report, sending...
I20260620 20:16:44.655157 8150 ts_manager.cc:194] Registered new tserver with Master: 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383)
I20260620 20:16:44.655725 8150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:38759
I20260620 20:16:44.664947 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:16:44.669106 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.665547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.24.20.190:33277)
0620 20:16:44.665701 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.665706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.665890 (+ 184us) server_negotiation.cc:407] Connection header received
0620 20:16:44.666086 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.666089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.666180 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.666293 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:44.666544 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.667035 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.667910 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.668122 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.668231 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.668536 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:44.668722 (+ 186us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 20:16:44.668980 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50810: token expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 20:16:44.669196 8341 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50810: token expired
W20260620 20:16:44.669446 8076 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.20.190:33277)
I20260620 20:16:44.676467 8341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.669794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.24.20.190:33277)
0620 20:16:44.669904 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.669910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.669925 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:16:44.670069 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.670072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.670111 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.670198 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:16:44.670781 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.671248 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.672018 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.672233 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.672339 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.673597 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:16:44.673600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:16:44.673603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:16:44.673634 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:16:44.674848 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.675293 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.675297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.675299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.675378 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:16:44.675659 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:16:44.675663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:16:44.675665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:16:44.675834 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:16:44.675973 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.676202 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.676313 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:16:44.678721 8150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:16:44.681936 8150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:16:44.689360 8273 tablet_service.cc:1467] Processing CreateTablet for tablet 9a686bbb7ced435fa11861913b6e6106 (DEFAULT_TABLE table=test-table [id=871b601b121d4d75be8d90aa614dfda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:44.690062 8273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a686bbb7ced435fa11861913b6e6106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.692912 8230 tablet_bootstrap.cc:492] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27: Bootstrap starting.
I20260620 20:16:44.693516 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.688349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:43383 (local address 127.0.0.1:49604)
0620 20:16:44.688505 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.688520 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.688596 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.689683 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.689686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.689701 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:44.689807 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.689813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.690980 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.690987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.692055 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.692064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.693103 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.693106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.693202 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.693246 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.693299 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.693374 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":9}
I20260620 20:16:44.693907 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.688546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:36257 (local address 127.0.0.1:55596)
0620 20:16:44.688847 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.688860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.688940 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.689804 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.689806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.689820 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:44.689909 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.689915 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.691180 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.691183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.692300 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.692310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.693477 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.693482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.693615 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.693649 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.693725 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.693787 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":113,"threads_started":1}
I20260620 20:16:44.694126 8230 tablet_bootstrap.cc:654] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.694499 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.688961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.24.20.130:36257)
0620 20:16:44.689339 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.689351 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.689378 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:16:44.689454 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.689460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.689660 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.689830 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:44.690006 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.691032 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.692428 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.693352 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.693454 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.694142 (+ 688us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.694245 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.694342 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":99,"threads_started":1}
I20260620 20:16:44.694741 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.688514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.24.20.131:43383)
0620 20:16:44.689196 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.689207 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.689229 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:16:44.689315 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.689324 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.689491 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.689959 (+ 468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:44.690143 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.690868 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.692171 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.692954 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.694207 (+ 1253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.694336 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.694379 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.694597 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":179,"threads_started":1}
I20260620 20:16:44.695569 8230 tablet_bootstrap.cc:492] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27: No bootstrap required, opened a new log
I20260620 20:16:44.695657 8230 ts_tablet_manager.cc:1397] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:16:44.696035 8230 raft_consensus.cc:348] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:44.696153 8230 raft_consensus.cc:374] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.696200 8230 raft_consensus.cc:729] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c601a64c90040c1bf94eee2bf5e0d27, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.696316 8230 consensus_queue.cc:260] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:44.696626 8230 ts_tablet_manager.cc:1428] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 20:16:44.697542 8545 tablet_service.cc:1467] Processing CreateTablet for tablet 9a686bbb7ced435fa11861913b6e6106 (DEFAULT_TABLE table=test-table [id=871b601b121d4d75be8d90aa614dfda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:44.697535 8412 tablet_service.cc:1467] Processing CreateTablet for tablet 9a686bbb7ced435fa11861913b6e6106 (DEFAULT_TABLE table=test-table [id=871b601b121d4d75be8d90aa614dfda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:16:44.698587 8412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a686bbb7ced435fa11861913b6e6106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.698589 8545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a686bbb7ced435fa11861913b6e6106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:16:44.706518 8625 tablet_bootstrap.cc:492] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e: Bootstrap starting.
I20260620 20:16:44.706950 8624 tablet_bootstrap.cc:492] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4: Bootstrap starting.
I20260620 20:16:44.709771 8625 tablet_bootstrap.cc:654] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.709800 8624 tablet_bootstrap.cc:654] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4: Neither blocks nor log segments found. Creating new log.
I20260620 20:16:44.710673 8624 log.cc:824] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:44.710673 8625 log.cc:824] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:16:44.712730 8625 tablet_bootstrap.cc:492] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e: No bootstrap required, opened a new log
I20260620 20:16:44.712730 8624 tablet_bootstrap.cc:492] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4: No bootstrap required, opened a new log
I20260620 20:16:44.712919 8625 ts_tablet_manager.cc:1397] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 20:16:44.712919 8624 ts_tablet_manager.cc:1397] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 20:16:44.716835 8624 raft_consensus.cc:348] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:44.716835 8625 raft_consensus.cc:348] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:44.717118 8625 raft_consensus.cc:374] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.717118 8624 raft_consensus.cc:374] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:16:44.717180 8624 raft_consensus.cc:729] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130e36847d074a8099c7306e54b194b4, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.717187 8625 raft_consensus.cc:729] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b332032d224af28ccbdca76055772e, State: Initialized, Role: FOLLOWER
I20260620 20:16:44.717653 8625 consensus_queue.cc:260] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:44.717654 8624 consensus_queue.cc:260] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:44.718300 8478 heartbeater.cc:503] Master 127.24.20.190:33277 was elected leader, sending a full tablet report...
I20260620 20:16:44.718501 8611 heartbeater.cc:503] Master 127.24.20.190:33277 was elected leader, sending a full tablet report...
I20260620 20:16:44.718652 8624 ts_tablet_manager.cc:1428] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:16:44.718652 8625 ts_tablet_manager.cc:1428] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 20:16:44.755537 8340 tablet.cc:2369] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:16:44.887821 8612 tablet.cc:2369] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:16:44.954497 8479 tablet.cc:2369] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:16:44.980381 8628 raft_consensus.cc:482] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:16:44.980598 8628 raft_consensus.cc:504] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:44.981823 8628 leader_election.cc:288] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383), 4c601a64c90040c1bf94eee2bf5e0d27 (127.24.20.129:39969)
I20260620 20:16:44.987300 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.982176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:52769 (local address 127.24.20.131:43383)
0620 20:16:44.982315 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.982321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.982568 (+ 247us) server_negotiation.cc:407] Connection header received
0620 20:16:44.982644 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.982647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.982698 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.982803 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:44.983183 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.983992 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.985870 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.986539 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.986677 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.986733 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.987104 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.987200 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:16:44.987303 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.982051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:39969 (local address 127.24.20.130:52177)
0620 20:16:44.982395 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.982411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.982513 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.982842 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.982845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.982855 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:44.983058 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.983065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.984105 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.984108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.985733 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.985746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.986679 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.986682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.986876 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.986939 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.987066 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.987147 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":250,"thread_start_us":83,"threads_started":1}
I20260620 20:16:44.987309 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.982176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:52177 (local address 127.24.20.129:39969)
0620 20:16:44.982444 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:44.982447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:44.982461 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:16:44.982645 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:44.982648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:44.982718 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:44.982830 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:16:44.983183 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.983992 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.985870 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.986587 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.986691 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.986840 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:44.987084 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 20:16:44.987160 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
I20260620 20:16:44.987676 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:44.982060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:43383 (local address 127.24.20.130:52769)
0620 20:16:44.982474 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:44.982485 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:16:44.982570 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:16:44.982831 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:16:44.982834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:16:44.982850 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:16:44.983058 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.983065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.984105 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.984108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.985733 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:16:44.985746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:16:44.986677 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:16:44.986681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:16:44.986875 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:44.986940 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 20:16:44.987437 (+ 497us) client_negotiation.cc:240] Negotiation successful
0620 20:16:44.987518 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":276,"spinlock_wait_cycles":6912,"thread_start_us":60,"threads_started":1}
I20260620 20:16:44.987907 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a686bbb7ced435fa11861913b6e6106" candidate_uuid: "03b332032d224af28ccbdca76055772e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" is_pre_election: true
I20260620 20:16:44.988250 8293 raft_consensus.cc:2454] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03b332032d224af28ccbdca76055772e in term 0.
I20260620 20:16:44.988307 8565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a686bbb7ced435fa11861913b6e6106" candidate_uuid: "03b332032d224af28ccbdca76055772e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130e36847d074a8099c7306e54b194b4" is_pre_election: true
I20260620 20:16:44.988585 8565 raft_consensus.cc:2454] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03b332032d224af28ccbdca76055772e in term 0.
I20260620 20:16:44.988754 8364 leader_election.cc:302] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b332032d224af28ccbdca76055772e, 4c601a64c90040c1bf94eee2bf5e0d27; no voters:
I20260620 20:16:44.989231 8628 raft_consensus.cc:2781] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:16:44.989301 8628 raft_consensus.cc:482] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:16:44.989331 8628 raft_consensus.cc:3037] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.990425 8628 raft_consensus.cc:504] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:44.990758 8628 leader_election.cc:288] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [CANDIDATE]: Term 1 election: Requested vote from peers 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383), 4c601a64c90040c1bf94eee2bf5e0d27 (127.24.20.129:39969)
I20260620 20:16:44.991186 8565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a686bbb7ced435fa11861913b6e6106" candidate_uuid: "03b332032d224af28ccbdca76055772e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130e36847d074a8099c7306e54b194b4"
I20260620 20:16:44.991186 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a686bbb7ced435fa11861913b6e6106" candidate_uuid: "03b332032d224af28ccbdca76055772e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c601a64c90040c1bf94eee2bf5e0d27"
I20260620 20:16:44.991310 8565 raft_consensus.cc:3037] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.991310 8293 raft_consensus.cc:3037] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:16:44.992553 8565 raft_consensus.cc:2454] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b332032d224af28ccbdca76055772e in term 1.
I20260620 20:16:44.992549 8293 raft_consensus.cc:2454] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b332032d224af28ccbdca76055772e in term 1.
I20260620 20:16:44.992882 8365 leader_election.cc:302] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b332032d224af28ccbdca76055772e, 130e36847d074a8099c7306e54b194b4; no voters:
I20260620 20:16:44.993083 8628 raft_consensus.cc:2781] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:16:44.993332 8628 raft_consensus.cc:686] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [term 1 LEADER]: Becoming Leader. State: Replica: 03b332032d224af28ccbdca76055772e, State: Running, Role: LEADER
I20260620 20:16:44.993662 8628 consensus_queue.cc:237] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:16:44.996867 8149 catalog_manager.cc:5495] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e reported cstate change: term changed from 0 to 1, leader changed from <none> to 03b332032d224af28ccbdca76055772e (127.24.20.130). New cstate: current_term: 1 leader_uuid: "03b332032d224af28ccbdca76055772e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } health_report { overall_health: UNKNOWN } } }
I20260620 20:16:45.058199 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.054814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.24.20.130:36257)
0620 20:16:45.054956 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 20:16:45.054960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:16:45.054972 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:16:45.055041 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:16:45.055044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:16:45.055104 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:16:45.055188 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:16:45.055428 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.055910 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.056941 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:16:45.057144 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:16:45.057220 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:16:45.057470 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:16:45.057827 (+ 357us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:16:45.057914 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:16:45.057971 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 20:16:45.058018 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:16:45.063742 8293 raft_consensus.cc:1261] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Refusing update from remote peer 03b332032d224af28ccbdca76055772e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:45.063745 8565 raft_consensus.cc:1261] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4 [term 1 FOLLOWER]: Refusing update from remote peer 03b332032d224af28ccbdca76055772e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:16:45.064481 8628 consensus_queue.cc:1035] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [LEADER]: Connected to new peer: Peer: permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:45.064741 8633 consensus_queue.cc:1035] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:16:45.072227 8636 mvcc.cc:205] Tried to move back new op lower bound from 7299017134333526016 to 7299017134056824832. Current Snapshot: MvccSnapshot[applied={T|T < 7299017134333526016 or (T in {7299017134333526016})}]
I20260620 20:16:45.078001 8149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:45.078171 8149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:16:45.080305 8149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Sending DeleteTablet for 3 replicas of tablet 9a686bbb7ced435fa11861913b6e6106
I20260620 20:16:45.080927 8545 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a686bbb7ced435fa11861913b6e6106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:45 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:49604
I20260620 20:16:45.081372 8646 tablet_replica.cc:330] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4: stopping tablet replica
I20260620 20:16:45.081643 8646 raft_consensus.cc:2229] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:16:45.081916 8646 raft_consensus.cc:2258] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:45.083029 8273 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a686bbb7ced435fa11861913b6e6106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:45 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:60758
I20260620 20:16:45.083272 8646 ts_tablet_manager.cc:1905] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:45.083729 8412 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a686bbb7ced435fa11861913b6e6106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:16:45 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:55596
I20260620 20:16:45.084432 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 8188
I20260620 20:16:45.087002 8646 ts_tablet_manager.cc:1918] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:45.087122 8646 log.cc:1196] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/9a686bbb7ced435fa11861913b6e6106
I20260620 20:16:45.087467 8646 ts_tablet_manager.cc:1939] T 9a686bbb7ced435fa11861913b6e6106 P 130e36847d074a8099c7306e54b194b4: Deleting consensus metadata
I20260620 20:16:45.087960 8648 tablet_replica.cc:330] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e: stopping tablet replica
I20260620 20:16:45.088311 8648 raft_consensus.cc:2229] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:16:45.088706 8135 catalog_manager.cc:4841] TS 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383): tablet 9a686bbb7ced435fa11861913b6e6106 (table test-table [id=871b601b121d4d75be8d90aa614dfda2]) successfully deleted
I20260620 20:16:45.089104 8648 raft_consensus.cc:2258] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:16:45.090431 8648 ts_tablet_manager.cc:1905] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:16:45.092555 8648 ts_tablet_manager.cc:1918] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:16:45.092691 8648 log.cc:1196] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/9a686bbb7ced435fa11861913b6e6106
I20260620 20:16:45.093019 8648 ts_tablet_manager.cc:1939] T 9a686bbb7ced435fa11861913b6e6106 P 03b332032d224af28ccbdca76055772e: Deleting consensus metadata
W20260620 20:16:45.093367 8134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:39969 (error 108)
I20260620 20:16:45.094113 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 8343
I20260620 20:16:45.094386 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.094110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:39969 (local address 127.0.0.1:60774)
0620 20:16:45.094217 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:45.094293 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:39969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 20:16:45.094921 8134 catalog_manager.cc:4584] TS 4c601a64c90040c1bf94eee2bf5e0d27 (127.24.20.129:39969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a686bbb7ced435fa11861913b6e6106: Network error: Client connection negotiation failed: client connection to 127.24.20.129:39969: connect: Connection refused (error 111)
I20260620 20:16:45.099558 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 8481
I20260620 20:16:45.100993 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:16:45.099784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:36257 (local address 127.0.0.1:55616)
0620 20:16:45.100835 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0620 20:16:45.100898 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.130:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":949}
I20260620 20:16:45.104974 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 8117
I20260620 20:17:05.111398 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:33277
--webserver_interface=127.24.20.190
--webserver_port=34581
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:33277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:05.222088 8650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:05.222340 8650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:05.222402 8650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:05.226632 8650 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:17:05.226701 8650 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:17:05.226725 8650 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:17:05.226748 8650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:05.226786 8650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:05.226831 8650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:05.226868 8650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:05.226904 8650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:05.231998 8650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:33277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:33277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=34581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:05.233027 8650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:05.233987 8650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:17:05.242422 8650 server_base.cc:1034] running on GCE node
W20260620 20:17:05.242412 8656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:05.242376 8658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:05.242638 8655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:05.243201 8650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:05.244246 8650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:05.245424 8650 hybrid_clock.cc:648] HybridClock initialized: now 1781986625245404 us; error 30 us; skew 500 ppm
Jun 20 20:17:05 dist-test-slave-pflr krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986625, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:05.248072 8650 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:17:05.248893 8650 webserver.cc:460] Webserver started at http://127.24.20.190:34581/ using document root <none> and password file <none>
I20260620 20:17:05.249410 8650 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:05.249470 8650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:05.253288 8650 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 20:17:05.255206 8665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:05.256395 8650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:05.256506 8650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "e88230dbd74442f6a7f9e53b69001873"
format_stamp: "Formatted at 2026-06-20 20:16:22 on dist-test-slave-pflr"
server_key: "9dd2dd2aae897c3cb90c226b3c700175"
server_key_iv: "7d4a0ab0f0ad299cab4e3c809569753e"
server_key_version: "encryptionkey@0"
I20260620 20:17:05.257257 8650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:05.269968 8650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:05.272683 8650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:05.272858 8650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:05.279657 8650 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:33277
I20260620 20:17:05.279949 8718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:33277 every 8 connection(s)
I20260620 20:17:05.280726 8650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:05.284812 8719 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:17:05.286348 8719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Bootstrap starting.
I20260620 20:17:05.287552 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 8650
I20260620 20:17:05.287962 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:39969
--local_ip_for_outbound_sockets=127.24.20.129
--tserver_master_addrs=127.24.20.190:33277
--webserver_port=42107
--webserver_interface=127.24.20.129
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:17:05.303182 8719 log.cc:824] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:05.314605 8719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:17:05.314980 8719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Bootstrap complete.
I20260620 20:17:05.318868 8719 raft_consensus.cc:348] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } }
I20260620 20:17:05.319216 8719 raft_consensus.cc:729] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e88230dbd74442f6a7f9e53b69001873, State: Initialized, Role: FOLLOWER
I20260620 20:17:05.319716 8719 consensus_queue.cc:260] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } }
I20260620 20:17:05.319829 8719 raft_consensus.cc:388] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:05.319907 8719 raft_consensus.cc:482] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:05.320020 8719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 2 FOLLOWER]: Advancing to term 3
I20260620 20:17:05.321853 8719 raft_consensus.cc:504] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } }
I20260620 20:17:05.322168 8719 leader_election.cc:302] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e88230dbd74442f6a7f9e53b69001873; no voters:
I20260620 20:17:05.322491 8719 leader_election.cc:288] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 20:17:05.322568 8723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 20:17:05.322762 8723 raft_consensus.cc:686] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [term 3 LEADER]: Becoming Leader. State: Replica: e88230dbd74442f6a7f9e53b69001873, State: Running, Role: LEADER
I20260620 20:17:05.323047 8723 consensus_queue.cc:237] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } }
I20260620 20:17:05.323660 8719 sys_catalog.cc:564] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:05.323910 8723 sys_catalog.cc:455] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e88230dbd74442f6a7f9e53b69001873. Latest consensus state: current_term: 3 leader_uuid: "e88230dbd74442f6a7f9e53b69001873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } } }
I20260620 20:17:05.324035 8723 sys_catalog.cc:458] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:05.324395 8724 sys_catalog.cc:455] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e88230dbd74442f6a7f9e53b69001873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e88230dbd74442f6a7f9e53b69001873" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33277 } } }
I20260620 20:17:05.324626 8724 sys_catalog.cc:458] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:05.324410 8727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:05.329540 8727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:05.330421 8727 catalog_manager.cc:1259] Loaded cluster ID: eed7674130714db094826ae2c66bbd30
I20260620 20:17:05.330480 8727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:05.333128 8727 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:05.333814 8727 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Loaded TSK: 0
I20260620 20:17:05.333904 8727 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e88230dbd74442f6a7f9e53b69001873: Loaded TSK: 1
F20260620 20:17:05.334576 8727 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6226a959ad google::LogMessage::Fail() at ??:0
@ 0x7f6226a9888a google::LogMessage::SendToLog() at ??:0
@ 0x7f6226a954de google::LogMessage::Flush() at ??:0
@ 0x7f6226a96a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f622975918d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6229751f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f62313fa7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f62313d499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f62313d5178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f6231416df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f623143582b std::function<>::operator()() at ??:0
@ 0x7f62313d4b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f62313d5f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f62313d2579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6231416640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6228ce73d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6228ce7c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6228ce93f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6228cd8512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f622a31c6db start_thread at ??:0
@ 0x7f622559361f clone at ??:0
W20260620 20:17:05.437239 8721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:05.437487 8721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:05.437559 8721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:05.441584 8721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:05.441633 8721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:05.441721 8721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:17:05.446995 8721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:39969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=42107
--tserver_master_addrs=127.24.20.190:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:05.447964 8721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:05.449007 8721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:05.459636 8770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:05.459636 8768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:05.459656 8767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:05.460667 8721 server_base.cc:1034] running on GCE node
I20260620 20:17:05.461082 8721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:05.461736 8721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:05.463020 8721 hybrid_clock.cc:648] HybridClock initialized: now 1781986625462984 us; error 52 us; skew 500 ppm
Jun 20 20:17:05 dist-test-slave-pflr krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986625, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:05.465828 8721 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:17:05.466689 8721 webserver.cc:460] Webserver started at http://127.24.20.129:42107/ using document root <none> and password file <none>
I20260620 20:17:05.467240 8721 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:05.467299 8721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:05.471256 8721 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:05.473652 8777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:05.475050 8721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:05.475178 8721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "4c601a64c90040c1bf94eee2bf5e0d27"
format_stamp: "Formatted at 2026-06-20 20:16:22 on dist-test-slave-pflr"
server_key: "ec58f4d05e919d8b52e83df3172af91f"
server_key_iv: "32297125c1903f8ea4f6a02cfcf8fae8"
server_key_version: "encryptionkey@0"
I20260620 20:17:05.475725 8721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:05.498811 8721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:05.502488 8721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:05.502700 8721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:05.503329 8721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:05.504573 8784 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 20:17:05.510465 8721 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 20:17:05.510576 8721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 20:17:05.510672 8721 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 20:17:05.512485 8721 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 20:17:05.512557 8721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 20:17:05.512898 8784 tablet_bootstrap.cc:492] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27: Bootstrap starting.
I20260620 20:17:05.523438 8721 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:39969
I20260620 20:17:05.523666 8891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:39969 every 8 connection(s)
I20260620 20:17:05.524516 8721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:17:05.524807 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 8721
I20260620 20:17:05.525345 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:36257
--local_ip_for_outbound_sockets=127.24.20.130
--tserver_master_addrs=127.24.20.190:33277
--webserver_port=43889
--webserver_interface=127.24.20.130
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:17:05.536628 8784 log.cc:824] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:05.545809 8784 tablet_bootstrap.cc:492] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:17:05.546270 8784 tablet_bootstrap.cc:492] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27: Bootstrap complete.
I20260620 20:17:05.547693 8784 ts_tablet_manager.cc:1397] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.014s
I20260620 20:17:05.550913 8784 raft_consensus.cc:348] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:17:05.551476 8784 raft_consensus.cc:729] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c601a64c90040c1bf94eee2bf5e0d27, State: Initialized, Role: FOLLOWER
I20260620 20:17:05.552026 8784 consensus_queue.cc:260] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:17:05.553068 8784 ts_tablet_manager.cc:1428] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27: Time spent starting tablet: real 0.005s user 0.006s sys 0.002s
W20260620 20:17:05.563699 8895 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:05.527660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:44341)
0620 20:17:05.528175 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:05.528218 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:05.529375 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:05.562992 (+ 33617us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":148,"threads_started":1}
W20260620 20:17:05.564160 8892 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:33277 (0 consecutive failures): Network error: Failed to ping master at 127.24.20.190:33277: Client connection negotiation failed: client connection to 127.24.20.190:33277: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:17:05.564877 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:05.564657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:59447)
0620 20:17:05.564769 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:05.564828 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 20:17:05.565517 8895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:05.565342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:60337)
0620 20:17:05.565428 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:05.565466 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 20:17:05.565709 8892 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:17:05.638156 8894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:05.638396 8894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:05.638438 8894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:05.642391 8894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:05.642439 8894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:05.642552 8894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:17:05.647683 8894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:36257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=43889
--tserver_master_addrs=127.24.20.190:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:05.648643 8894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:05.649598 8894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:05.658854 8903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:05.658871 8902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:05.658867 8905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:05.659127 8894 server_base.cc:1034] running on GCE node
I20260620 20:17:05.659595 8894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:05.660283 8894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:05.661448 8894 hybrid_clock.cc:648] HybridClock initialized: now 1781986625661406 us; error 53 us; skew 500 ppm
Jun 20 20:17:05 dist-test-slave-pflr krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986625, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:05.664165 8894 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:17:05.665045 8894 webserver.cc:460] Webserver started at http://127.24.20.130:43889/ using document root <none> and password file <none>
I20260620 20:17:05.665588 8894 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:05.665678 8894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:05.669536 8894 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:05.671584 8912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:05.673023 8894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:05.673177 8894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "03b332032d224af28ccbdca76055772e"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-pflr"
server_key: "1b159e431479774825754c44ae24f130"
server_key_iv: "cf4ca4c09318dd893e5f83f686ea113e"
server_key_version: "encryptionkey@0"
I20260620 20:17:05.673753 8894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:05.686203 8894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:05.689342 8894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:05.689548 8894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:05.690133 8894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:05.690979 8894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:05.691053 8894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:05.691124 8894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:05.691175 8894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:05.700538 8894 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:36257
I20260620 20:17:05.700659 9025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:36257 every 8 connection(s)
I20260620 20:17:05.701609 8894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:17:05.702770 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 8894
I20260620 20:17:05.703212 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:43383
--local_ip_for_outbound_sockets=127.24.20.131
--tserver_master_addrs=127.24.20.190:33277
--webserver_port=46521
--webserver_interface=127.24.20.131
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:40015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:17:05.705915 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:05.703881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:39755)
0620 20:17:05.704421 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:05.704589 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
W20260620 20:17:05.706338 9026 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:33277 (0 consecutive failures): Network error: Failed to ping master at 127.24.20.190:33277: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
I20260620 20:17:05.707073 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:05.706761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:60193)
0620 20:17:05.706913 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:05.706982 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 20:17:05.707967 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:05.707656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:40877)
0620 20:17:05.707800 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:05.707878 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 20:17:05.708163 9026 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:17:05.815599 9029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:05.815841 9029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:05.815887 9029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:05.819754 9029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:05.819810 9029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:05.819866 9029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:17:05.824970 9029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:40015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:43383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=46521
--tserver_master_addrs=127.24.20.190:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:05.825888 9029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:05.826825 9029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:05.836069 9035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:05.836037 9037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:05.836047 9034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:05.836247 9029 server_base.cc:1034] running on GCE node
I20260620 20:17:05.836763 9029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:05.837409 9029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:05.838613 9029 hybrid_clock.cc:648] HybridClock initialized: now 1781986625838597 us; error 35 us; skew 500 ppm
Jun 20 20:17:05 dist-test-slave-pflr krb5kdc[7566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986625, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:05.841256 9029 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:17:05.842104 9029 webserver.cc:460] Webserver started at http://127.24.20.131:46521/ using document root <none> and password file <none>
I20260620 20:17:05.842661 9029 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:05.842747 9029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:05.846611 9029 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:05.848687 9044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:05.849916 9029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:05.850041 9029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "130e36847d074a8099c7306e54b194b4"
format_stamp: "Formatted at 2026-06-20 20:16:23 on dist-test-slave-pflr"
server_key: "dd89a9c0703dbf72c7e27a6e86b08d0a"
server_key_iv: "d730f9c9d103ec5947dc61a09f8a48bc"
server_key_version: "encryptionkey@0"
I20260620 20:17:05.850605 9029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:05.866149 9029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:05.869540 9029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:05.869721 9029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:05.870318 9029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:05.871129 9029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:05.871196 9029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:05.871260 9029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:05.871306 9029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:05.880313 9029 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:43383
I20260620 20:17:05.880508 9157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:43383 every 8 connection(s)
I20260620 20:17:05.881386 9029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:17:05.885131 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:05.883379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:40485)
0620 20:17:05.883826 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:05.883937 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":121,"threads_started":1}
W20260620 20:17:05.885673 9158 heartbeater.cc:644] Failed to heartbeat to 127.24.20.190:33277 (0 consecutive failures): Network error: Failed to ping master at 127.24.20.190:33277: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
I20260620 20:17:05.886227 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:05.886030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:50353)
0620 20:17:05.886129 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:05.886175 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 20:17:05.886821 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:05.886643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:33047)
0620 20:17:05.886735 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:05.886776 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 20:17:05.887034 9158 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:17:05.890506 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 9029
I20260620 20:17:06.566635 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:06.566254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:35793)
0620 20:17:06.566475 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:06.566526 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":71,"threads_started":1}
I20260620 20:17:06.709180 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:06.708832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:57775)
0620 20:17:06.709036 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:06.709086 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":60,"threads_started":1}
I20260620 20:17:06.887918 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:06.887513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:43439)
0620 20:17:06.887768 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:06.887818 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":87,"threads_started":1}
I20260620 20:17:07.115440 9165 raft_consensus.cc:482] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:07.115706 9165 raft_consensus.cc:504] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:17:07.116850 9165 leader_election.cc:288] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383), 03b332032d224af28ccbdca76055772e (127.24.20.130:36257)
Jun 20 20:17:07 dist-test-slave-pflr krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986625, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.130@KRBTEST.COM
Jun 20 20:17:07 dist-test-slave-pflr krb5kdc[7566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986625, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.131@KRBTEST.COM
I20260620 20:17:07.130185 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:07.117100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:36257 (local address 127.24.20.129:53829)
0620 20:17:07.117390 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:07.117403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:07.117505 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:07.118972 (+ 1467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:07.118977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:07.119511 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:07.119941 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:07.119956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.121210 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.121213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:07.122046 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:07.122055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.122739 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.122743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:07.122916 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:07.123550 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:07.123565 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:07.125189 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:07.127533 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:07.127538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:07.127552 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:07.127979 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:07.128499 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:07.128501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:07.128503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:07.128573 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:07.129184 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:07.129186 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:07.129402 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 20:17:07.129626 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 20:17:07.129971 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":93,"spinlock_wait_cycles":18176,"thread_start_us":69,"threads_started":1}
I20260620 20:17:07.130184 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:07.117030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:43383 (local address 127.24.20.129:39553)
0620 20:17:07.117346 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:07.117360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:07.117468 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:07.118972 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:07.118978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:07.119511 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:07.119942 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:07.119956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.121216 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.121221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:07.122042 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:07.122053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.122736 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.122740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:07.122914 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:07.123618 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:07.123630 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:07.125516 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:07.127578 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:07.127580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:07.127582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:07.127979 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:07.128268 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:07.128270 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:07.128271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:07.128387 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:07.129113 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:07.129117 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:07.129402 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 20:17:07.129626 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 20:17:07.130020 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":218,"spinlock_wait_cycles":26240,"thread_start_us":75,"threads_started":1}
I20260620 20:17:07.130241 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:07.117267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:53829 (local address 127.24.20.130:36257)
0620 20:17:07.117647 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:07.117664 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:07.117693 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:17:07.118304 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:07.118309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:07.118638 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:07.118930 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:07.120093 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.121084 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.122181 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.122627 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.122786 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:07.125396 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:07.125400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:07.125411 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:07.125436 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:07.127420 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:07.128295 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:07.128299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:07.128302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:07.128396 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:07.128682 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:07.128685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:07.128687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:07.129095 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:07.129203 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:07.129781 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 20:17:07.130055 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":65,"threads_started":1}
I20260620 20:17:07.131106 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:07.117194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:39553 (local address 127.24.20.131:43383)
0620 20:17:07.117586 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:07.117615 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:07.117650 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:17:07.118305 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:07.118310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:07.118638 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:07.118930 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:07.120111 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.121096 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.122160 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:07.122617 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:07.122786 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:07.125694 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:07.125702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:07.125717 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:07.125757 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:07.127498 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:07.128079 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:07.128083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:07.128087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:07.128190 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:07.128483 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:07.128489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:07.128493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:07.129004 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:07.129147 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:07.129781 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 20:17:07.130055 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":58,"threads_started":1}
W20260620 20:17:07.132715 8781 leader_election.cc:341] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03b332032d224af28ccbdca76055772e (127.24.20.130:36257): Not found: Tablet not found: 9a686bbb7ced435fa11861913b6e6106
W20260620 20:17:07.132967 8779 leader_election.cc:341] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383): Not found: Tablet not found: 9a686bbb7ced435fa11861913b6e6106
I20260620 20:17:07.133062 8779 leader_election.cc:302] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c601a64c90040c1bf94eee2bf5e0d27; no voters: 03b332032d224af28ccbdca76055772e, 130e36847d074a8099c7306e54b194b4
I20260620 20:17:07.133309 9165 raft_consensus.cc:2726] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:17:07.567651 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:07.567386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:34253)
0620 20:17:07.567513 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:07.567570 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 20:17:07.710260 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:07.709897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:41697)
0620 20:17:07.710125 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:07.710177 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":67,"threads_started":1}
I20260620 20:17:07.888967 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:07.888592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:52641)
0620 20:17:07.888835 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:07.888883 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
I20260620 20:17:08.568794 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:08.568353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:39239)
0620 20:17:08.568584 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:08.568688 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260620 20:17:08.711444 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:08.711053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:36083)
0620 20:17:08.711302 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:08.711356 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
I20260620 20:17:08.889979 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:08.889636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:38495)
0620 20:17:08.889850 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:08.889898 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20260620 20:17:09.151185 9175 raft_consensus.cc:482] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:09.151322 9175 raft_consensus.cc:504] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:17:09.151774 9175 leader_election.cc:288] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383), 03b332032d224af28ccbdca76055772e (127.24.20.130:36257)
W20260620 20:17:09.152642 8779 leader_election.cc:341] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383): Not found: Tablet not found: 9a686bbb7ced435fa11861913b6e6106
W20260620 20:17:09.152742 8781 leader_election.cc:341] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03b332032d224af28ccbdca76055772e (127.24.20.130:36257): Not found: Tablet not found: 9a686bbb7ced435fa11861913b6e6106
I20260620 20:17:09.152851 8781 leader_election.cc:302] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c601a64c90040c1bf94eee2bf5e0d27; no voters: 03b332032d224af28ccbdca76055772e, 130e36847d074a8099c7306e54b194b4
I20260620 20:17:09.153038 9175 raft_consensus.cc:2726] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:17:09.569855 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:09.569454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:53047)
0620 20:17:09.569705 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:09.569759 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
I20260620 20:17:09.712492 9177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:09.712124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:51625)
0620 20:17:09.712362 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:09.712410 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260620 20:17:09.890977 9178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:09.890609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:49817)
0620 20:17:09.890836 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:09.890887 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260620 20:17:10.571337 9179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:10.570843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:45399)
0620 20:17:10.571173 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:10.571240 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":76,"threads_started":1}
I20260620 20:17:10.713819 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:10.713377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:57007)
0620 20:17:10.713658 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:10.713723 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":74,"threads_started":1}
I20260620 20:17:10.892191 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:10.891767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:37297)
0620 20:17:10.892021 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:10.892086 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260620 20:17:10.902890 9182 raft_consensus.cc:482] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:10.903016 9182 raft_consensus.cc:504] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:17:10.903486 9182 leader_election.cc:288] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383), 03b332032d224af28ccbdca76055772e (127.24.20.130:36257)
W20260620 20:17:10.904238 8779 leader_election.cc:341] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383): Not found: Tablet not found: 9a686bbb7ced435fa11861913b6e6106
W20260620 20:17:10.904325 8781 leader_election.cc:341] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03b332032d224af28ccbdca76055772e (127.24.20.130:36257): Not found: Tablet not found: 9a686bbb7ced435fa11861913b6e6106
I20260620 20:17:10.904433 8781 leader_election.cc:302] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c601a64c90040c1bf94eee2bf5e0d27; no voters: 03b332032d224af28ccbdca76055772e, 130e36847d074a8099c7306e54b194b4
I20260620 20:17:10.904614 9182 raft_consensus.cc:2726] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:17:11.572634 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:11.572167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:43109)
0620 20:17:11.572439 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:11.572494 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
I20260620 20:17:11.714983 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:11.714607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:48333)
0620 20:17:11.714843 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:11.714894 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":64,"threads_started":1}
I20260620 20:17:11.893280 9185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:11.892921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:40659)
0620 20:17:11.893144 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:11.893195 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260620 20:17:12.573740 9186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:12.573360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:44667)
0620 20:17:12.573593 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:12.573644 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260620 20:17:12.716125 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:12.715679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:60873)
0620 20:17:12.715957 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:12.716010 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":68,"threads_started":1}
I20260620 20:17:12.894356 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:12.893932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:33699)
0620 20:17:12.894212 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:12.894264 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":64,"threads_started":1}
I20260620 20:17:13.574815 9189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:13.574421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:53109)
0620 20:17:13.574673 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:13.574724 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260620 20:17:13.717382 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:13.716911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:38915)
0620 20:17:13.717194 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:13.717258 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260620 20:17:13.895392 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:13.895013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:42203)
0620 20:17:13.895257 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:13.895308 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260620 20:17:14.576005 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:14.575613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:43877)
0620 20:17:14.575848 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:14.575902 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260620 20:17:14.718453 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:14.718077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:51387)
0620 20:17:14.718317 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:14.718369 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260620 20:17:14.896489 9194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:14.896097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:53875)
0620 20:17:14.896351 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:14.896401 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260620 20:17:15.577461 9195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:15.576983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:37023)
0620 20:17:15.577256 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:15.577316 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":105,"threads_started":1}
I20260620 20:17:15.719667 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:15.719254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:57841)
0620 20:17:15.719505 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:15.719575 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260620 20:17:15.774977 9197 raft_consensus.cc:482] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:15.775137 9197 raft_consensus.cc:504] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:17:15.775606 9197 leader_election.cc:288] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383), 03b332032d224af28ccbdca76055772e (127.24.20.130:36257)
W20260620 20:17:15.776270 8779 leader_election.cc:341] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383): Not found: Tablet not found: 9a686bbb7ced435fa11861913b6e6106
W20260620 20:17:15.776388 8781 leader_election.cc:341] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03b332032d224af28ccbdca76055772e (127.24.20.130:36257): Not found: Tablet not found: 9a686bbb7ced435fa11861913b6e6106
I20260620 20:17:15.776479 8781 leader_election.cc:302] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c601a64c90040c1bf94eee2bf5e0d27; no voters: 03b332032d224af28ccbdca76055772e, 130e36847d074a8099c7306e54b194b4
I20260620 20:17:15.776672 9197 raft_consensus.cc:2726] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:17:15.897596 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:15.897216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:56413)
0620 20:17:15.897449 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:15.897498 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
I20260620 20:17:16.578701 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:16.578199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:55263)
0620 20:17:16.578486 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:16.578595 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":68,"threads_started":1}
I20260620 20:17:16.720736 9200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:16.720315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:55989)
0620 20:17:16.720546 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:16.720649 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
I20260620 20:17:16.898687 9201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:16.898282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:60019)
0620 20:17:16.898550 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:16.898600 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":82,"threads_started":1}
I20260620 20:17:17.579787 9202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.579349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:52983)
0620 20:17:17.579606 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:17.579657 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260620 20:17:17.721832 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.721412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:59139)
0620 20:17:17.721692 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:17.721742 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":82,"threads_started":1}
I20260620 20:17:17.899719 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:17.899320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:50539)
0620 20:17:17.899583 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:17.899633 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":110,"threads_started":1}
I20260620 20:17:18.389853 9205 raft_consensus.cc:482] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:18.389981 9205 raft_consensus.cc:504] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130e36847d074a8099c7306e54b194b4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 43383 } } peers { permanent_uuid: "03b332032d224af28ccbdca76055772e" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 36257 } } peers { permanent_uuid: "4c601a64c90040c1bf94eee2bf5e0d27" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 39969 } }
I20260620 20:17:18.390424 9205 leader_election.cc:288] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383), 03b332032d224af28ccbdca76055772e (127.24.20.130:36257)
W20260620 20:17:18.391136 8779 leader_election.cc:341] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 130e36847d074a8099c7306e54b194b4 (127.24.20.131:43383): Not found: Tablet not found: 9a686bbb7ced435fa11861913b6e6106
W20260620 20:17:18.391223 8781 leader_election.cc:341] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 03b332032d224af28ccbdca76055772e (127.24.20.130:36257): Not found: Tablet not found: 9a686bbb7ced435fa11861913b6e6106
I20260620 20:17:18.391332 8781 leader_election.cc:302] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c601a64c90040c1bf94eee2bf5e0d27; no voters: 03b332032d224af28ccbdca76055772e, 130e36847d074a8099c7306e54b194b4
I20260620 20:17:18.391461 9205 raft_consensus.cc:2726] T 9a686bbb7ced435fa11861913b6e6106 P 4c601a64c90040c1bf94eee2bf5e0d27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:17:18.580853 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:18.580436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:56035)
0620 20:17:18.580710 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:18.580760 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":72,"threads_started":1}
I20260620 20:17:18.723093 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:18.722689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:52109)
0620 20:17:18.722935 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:18.722987 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260620 20:17:18.900935 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:18.900501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:42753)
0620 20:17:18.900793 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:18.900844 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":136,"threads_started":1}
I20260620 20:17:19.581856 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:19.581485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:59543)
0620 20:17:19.581709 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:19.581762 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
I20260620 20:17:19.724135 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:19.723764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:44931)
0620 20:17:19.723997 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:19.724046 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260620 20:17:19.901999 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:19.901624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:41591)
0620 20:17:19.901866 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:19.901913 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
I20260620 20:17:20.583047 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:20.582655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.129:33123)
0620 20:17:20.582901 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:20.582952 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260620 20:17:20.725473 9213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:20.725075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.130:43183)
0620 20:17:20.725329 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:20.725381 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260620 20:17:20.892211 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 8721
I20260620 20:17:20.897840 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 8894
I20260620 20:17:20.903187 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 9029
I20260620 20:17:20.903385 9214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:20.902894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:33277 (local address 127.24.20.131:54959)
0620 20:17:20.903183 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:20.903252 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.190:33277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
2026-06-20T20:17:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59449 ms)
[----------] 30 tests from SecurityITest (235268 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:20 dist-test-slave-pflr krb5kdc[9217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:20 dist-test-slave-pflr krb5kdc[9217](info): set up 2 sockets
Jun 20 20:17:20 dist-test-slave-pflr krb5kdc[9217](info): commencing operation
krb5kdc: starting...
W20260620 20:17:21.941260 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:22 dist-test-slave-pflr krb5kdc[9217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:17:22.065797 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:41505
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:41505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:22.178974 9230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:22.179306 9230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:22.179390 9230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:22.183715 9230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:22.183820 9230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:22.183856 9230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:22.183897 9230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:22.183926 9230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:22.189368 9230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:41505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:41505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:22.190455 9230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:22.191512 9230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:22.200307 9236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:22.200307 9235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:22.200419 9230 server_base.cc:1034] running on GCE node
W20260620 20:17:22.200340 9238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:22.201086 9230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:22.202171 9230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:22.203354 9230 hybrid_clock.cc:648] HybridClock initialized: now 1781986642203344 us; error 32 us; skew 500 ppm
Jun 20 20:17:22 dist-test-slave-pflr krb5kdc[9217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986642, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:22.206101 9230 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:17:22.206897 9230 webserver.cc:460] Webserver started at http://127.24.20.190:38163/ using document root <none> and password file <none>
I20260620 20:17:22.207450 9230 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:22.207540 9230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:22.207763 9230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:22.209770 9230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "9115992f7e38466b9ed3b96137a76a7f"
format_stamp: "Formatted at 2026-06-20 20:17:22 on dist-test-slave-pflr"
server_key: "be36e183f9d3598d7735b61f6c2361f6"
server_key_iv: "f62aebeab5f8935b9ba905b4e0e0e602"
server_key_version: "encryptionkey@0"
I20260620 20:17:22.210317 9230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "9115992f7e38466b9ed3b96137a76a7f"
format_stamp: "Formatted at 2026-06-20 20:17:22 on dist-test-slave-pflr"
server_key: "be36e183f9d3598d7735b61f6c2361f6"
server_key_iv: "f62aebeab5f8935b9ba905b4e0e0e602"
server_key_version: "encryptionkey@0"
I20260620 20:17:22.213785 9230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:22.215981 9245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:22.217211 9230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:17:22.217343 9230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "9115992f7e38466b9ed3b96137a76a7f"
format_stamp: "Formatted at 2026-06-20 20:17:22 on dist-test-slave-pflr"
server_key: "be36e183f9d3598d7735b61f6c2361f6"
server_key_iv: "f62aebeab5f8935b9ba905b4e0e0e602"
server_key_version: "encryptionkey@0"
I20260620 20:17:22.217447 9230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:22.240327 9230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:22.243953 9230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:22.244158 9230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:22.251116 9230 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:41505
I20260620 20:17:22.251150 9297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:41505 every 8 connection(s)
I20260620 20:17:22.251993 9230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:22.254688 9298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:22.259866 9298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f: Bootstrap starting.
I20260620 20:17:22.262291 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 9230
I20260620 20:17:22.262476 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:22.262693 9298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:22.262811 24658 external_mini_cluster.cc:1428] Setting key 941ccba9d3f973a75d1f9c3546094bdc
I20260620 20:17:22.263590 9298 log.cc:824] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:22.265570 9298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f: No bootstrap required, opened a new log
I20260620 20:17:22.269018 9298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115992f7e38466b9ed3b96137a76a7f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41505 } }
I20260620 20:17:22.269253 9298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:22.269337 9298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9115992f7e38466b9ed3b96137a76a7f, State: Initialized, Role: FOLLOWER
I20260620 20:17:22.269804 9298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115992f7e38466b9ed3b96137a76a7f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41505 } }
I20260620 20:17:22.269933 9298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:22.270016 9298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:22.270109 9298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:22.270956 9298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115992f7e38466b9ed3b96137a76a7f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41505 } }
Jun 20 20:17:22 dist-test-slave-pflr krb5kdc[9217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:17:22.271322 9298 leader_election.cc:302] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9115992f7e38466b9ed3b96137a76a7f; no voters:
I20260620 20:17:22.271659 9298 leader_election.cc:288] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:22.271721 9303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:22.271929 9303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [term 1 LEADER]: Becoming Leader. State: Replica: 9115992f7e38466b9ed3b96137a76a7f, State: Running, Role: LEADER
I20260620 20:17:22.272280 9303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115992f7e38466b9ed3b96137a76a7f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41505 } }
I20260620 20:17:22.272923 9298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:22.273965 9304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9115992f7e38466b9ed3b96137a76a7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115992f7e38466b9ed3b96137a76a7f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41505 } } }
I20260620 20:17:22.274101 9304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:22.274402 9305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9115992f7e38466b9ed3b96137a76a7f. Latest consensus state: current_term: 1 leader_uuid: "9115992f7e38466b9ed3b96137a76a7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9115992f7e38466b9ed3b96137a76a7f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41505 } } }
I20260620 20:17:22.274596 9305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:22.277770 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.263908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.24.20.190:41505)
0620 20:17:22.264360 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:22.264391 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:22.264437 (+ 46us) server_negotiation.cc:407] Connection header received
0620 20:17:22.265106 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:22.265123 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:22.265428 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:22.265755 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:22.266562 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.267499 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.268390 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.268905 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.269076 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.271829 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:22.271835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:22.271852 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:22.271889 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:22.274061 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:22.274706 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:22.274713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:22.274719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:22.275224 (+ 505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:22.275634 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:22.275637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:22.275639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:22.276110 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:22.276295 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:22.276570 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 20:17:22.276877 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":146,"threads_started":1}
W20260620 20:17:22.278038 9319 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:17:22.278106 9319 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:17:22.278172 9312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:22.278793 9312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:22.283701 9312 catalog_manager.cc:1347] Generated new cluster ID: ac58ced209554842af2af686fcb937a3
I20260620 20:17:22.283758 9312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:22.297259 9312 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:22.298395 9312 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:22.304661 9312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f: Generated new TSK 0
I20260620 20:17:22.305425 9312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:17:22.341677 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41505
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:22.450330 9324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:22.450603 9324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:22.450670 9324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:22.454668 9324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:22.454751 9324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:22.454841 9324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:17:22.460001 9324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:22.461068 9324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:22.462041 9324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:22.471397 9329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:22.471408 9330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:22.471503 9324 server_base.cc:1034] running on GCE node
W20260620 20:17:22.471402 9332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:22.472019 9324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:22.472762 9324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:22.473950 9324 hybrid_clock.cc:648] HybridClock initialized: now 1781986642473931 us; error 30 us; skew 500 ppm
Jun 20 20:17:22 dist-test-slave-pflr krb5kdc[9217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986642, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:22.476696 9324 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:17:22.477487 9324 webserver.cc:460] Webserver started at http://127.24.20.129:44313/ using document root <none> and password file <none>
I20260620 20:17:22.478015 9324 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:22.478121 9324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:22.478338 9324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:22.480312 9324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "a4ac0dabf130465e9512ee7ccdefaa64"
format_stamp: "Formatted at 2026-06-20 20:17:22 on dist-test-slave-pflr"
server_key: "66637f092f9f040d8f02cb8c7ceffe21"
server_key_iv: "71909b68135ee559c110911041545982"
server_key_version: "encryptionkey@0"
I20260620 20:17:22.480870 9324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "a4ac0dabf130465e9512ee7ccdefaa64"
format_stamp: "Formatted at 2026-06-20 20:17:22 on dist-test-slave-pflr"
server_key: "66637f092f9f040d8f02cb8c7ceffe21"
server_key_iv: "71909b68135ee559c110911041545982"
server_key_version: "encryptionkey@0"
I20260620 20:17:22.484424 9324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:17:22.486713 9339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:22.487943 9324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:22.488081 9324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "a4ac0dabf130465e9512ee7ccdefaa64"
format_stamp: "Formatted at 2026-06-20 20:17:22 on dist-test-slave-pflr"
server_key: "66637f092f9f040d8f02cb8c7ceffe21"
server_key_iv: "71909b68135ee559c110911041545982"
server_key_version: "encryptionkey@0"
I20260620 20:17:22.488196 9324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:22.509392 9324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:22.512450 9324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:22.512689 9324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:22.513283 9324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:22.514099 9324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:22.514169 9324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:22.514240 9324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:22.514282 9324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:22.523330 9324 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:34645
I20260620 20:17:22.523355 9452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:34645 every 8 connection(s)
I20260620 20:17:22.524181 9324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:17:22.527679 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 9324
I20260620 20:17:22.527823 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:17:22.528151 24658 external_mini_cluster.cc:1428] Setting key 4c49552305b52e27a528e1a656c5d40b
Jun 20 20:17:22 dist-test-slave-pflr krb5kdc[9217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986642, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:17:22.538291 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.526046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:58463 (local address 127.24.20.190:41505)
0620 20:17:22.526210 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:22.526215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:22.527028 (+ 813us) server_negotiation.cc:407] Connection header received
0620 20:17:22.528474 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:22.528478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:22.528545 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:22.528675 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:22.530015 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.530546 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.531684 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.531917 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.532031 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.534491 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:22.534494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:22.534497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:22.534528 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:22.536096 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:22.536705 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:22.536711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:22.536713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:22.536801 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:22.537202 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:22.537205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:22.537208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:22.537449 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:22.537534 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:22.538066 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 20:17:22.538168 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:17:22.539243 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.526451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:41505 (local address 127.24.20.129:58463)
0620 20:17:22.526874 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:22.526915 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:22.528256 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:22.528865 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:22.528872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:22.529342 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:22.529869 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:22.529883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.530677 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.530681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:22.531554 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:22.531564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.532066 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.532070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:22.532237 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.532898 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:22.532914 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:22.534327 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:22.536260 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:22.536266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:22.536281 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:22.536569 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:22.536913 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:22.536917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:22.536919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:22.537068 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:22.537549 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:22.537554 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:22.537788 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 20:17:22.537963 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 20:17:22.538302 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
I20260620 20:17:22.540701 9453 heartbeater.cc:347] Connected to a master server at 127.24.20.190:41505
I20260620 20:17:22.540972 9453 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:22.541479 9453 heartbeater.cc:511] Master 127.24.20.190:41505 requested a full tablet report, sending...
I20260620 20:17:22.543038 9262 ts_manager.cc:194] Registered new tserver with Master: a4ac0dabf130465e9512ee7ccdefaa64 (127.24.20.129:34645)
I20260620 20:17:22.544232 9262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:58463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:17:22.555785 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41505
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:22.660748 9458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:22.661005 9458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:22.661080 9458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:22.665045 9458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:22.665118 9458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:22.665202 9458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:17:22.670200 9458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:22.671145 9458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:22.672102 9458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:22.681418 9464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:22.681402 9466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:22.681397 9463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:22.681636 9458 server_base.cc:1034] running on GCE node
I20260620 20:17:22.681995 9458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:22.682699 9458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:22.683887 9458 hybrid_clock.cc:648] HybridClock initialized: now 1781986642683867 us; error 51 us; skew 500 ppm
Jun 20 20:17:22 dist-test-slave-pflr krb5kdc[9217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986642, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:22.686465 9458 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:17:22.687295 9458 webserver.cc:460] Webserver started at http://127.24.20.130:35869/ using document root <none> and password file <none>
I20260620 20:17:22.687822 9458 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:22.687906 9458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:22.688124 9458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:22.690060 9458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "2abbdd9e21104c18b51922038e6aa5f2"
format_stamp: "Formatted at 2026-06-20 20:17:22 on dist-test-slave-pflr"
server_key: "ff226755b0eac085680b8e396bc21ceb"
server_key_iv: "db8c85bad31c18bdafc3a0bfd3989f23"
server_key_version: "encryptionkey@0"
I20260620 20:17:22.690593 9458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "2abbdd9e21104c18b51922038e6aa5f2"
format_stamp: "Formatted at 2026-06-20 20:17:22 on dist-test-slave-pflr"
server_key: "ff226755b0eac085680b8e396bc21ceb"
server_key_iv: "db8c85bad31c18bdafc3a0bfd3989f23"
server_key_version: "encryptionkey@0"
I20260620 20:17:22.694180 9458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 20:17:22.696440 9473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:22.697635 9458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:17:22.697769 9458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "2abbdd9e21104c18b51922038e6aa5f2"
format_stamp: "Formatted at 2026-06-20 20:17:22 on dist-test-slave-pflr"
server_key: "ff226755b0eac085680b8e396bc21ceb"
server_key_iv: "db8c85bad31c18bdafc3a0bfd3989f23"
server_key_version: "encryptionkey@0"
I20260620 20:17:22.697875 9458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:22.715747 9458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:22.718516 9458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:22.718717 9458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:22.719250 9458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:22.720052 9458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:22.720129 9458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:22.720193 9458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:22.720244 9458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:22.729056 9458 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:44127
I20260620 20:17:22.729105 9586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:44127 every 8 connection(s)
I20260620 20:17:22.729907 9458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:17:22.731521 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 9458
I20260620 20:17:22.731670 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:17:22.731953 24658 external_mini_cluster.cc:1428] Setting key d5084d7f9ac0eaaf4221a41341e836c1
Jun 20 20:17:22 dist-test-slave-pflr krb5kdc[9217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986642, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
I20260620 20:17:22.744048 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.731695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:41903 (local address 127.24.20.190:41505)
0620 20:17:22.731841 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:22.731846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:22.732784 (+ 938us) server_negotiation.cc:407] Connection header received
0620 20:17:22.734575 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:22.734578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:22.734619 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:22.734687 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:22.735986 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.736558 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.737812 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.738041 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.738153 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.740514 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:22.740519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:22.740522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:22.740546 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:22.742089 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:22.742585 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:22.742588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:22.742589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:22.742661 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:22.743025 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:22.743027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:22.743028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:22.743206 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:22.743281 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:22.743828 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 20:17:22.743931 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:17:22.745204 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.732089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:41505 (local address 127.24.20.130:41903)
0620 20:17:22.732626 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:22.732681 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:22.734350 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:22.734854 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:22.734865 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:22.735369 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:22.735842 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:22.735854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.736721 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.736725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:22.737693 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:22.737702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.738185 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.738189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:22.738373 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.738958 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:22.738974 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:22.740340 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:22.742192 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:22.742197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:22.742207 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:22.742484 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:22.742743 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:22.742745 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:22.742746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:22.742866 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:22.743294 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:22.743299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:22.743576 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 20:17:22.743752 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 20:17:22.744102 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":141,"threads_started":1}
I20260620 20:17:22.746454 9587 heartbeater.cc:347] Connected to a master server at 127.24.20.190:41505
I20260620 20:17:22.746680 9587 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:22.747128 9587 heartbeater.cc:511] Master 127.24.20.190:41505 requested a full tablet report, sending...
I20260620 20:17:22.748132 9262 ts_manager.cc:194] Registered new tserver with Master: 2abbdd9e21104c18b51922038e6aa5f2 (127.24.20.130:44127)
I20260620 20:17:22.748750 9262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:41903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:17:22.759371 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41505
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:22.866964 9592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:22.867200 9592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:22.867240 9592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:22.871225 9592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:22.871277 9592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:22.871332 9592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:17:22.876502 9592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:22.877449 9592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:22.878408 9592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:22.887567 9600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:22.887575 9598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:22.887549 9597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:22.887770 9592 server_base.cc:1034] running on GCE node
I20260620 20:17:22.888235 9592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:22.888841 9592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:22.889991 9592 hybrid_clock.cc:648] HybridClock initialized: now 1781986642889965 us; error 41 us; skew 500 ppm
Jun 20 20:17:22 dist-test-slave-pflr krb5kdc[9217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986642, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:22.892765 9592 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:17:22.893532 9592 webserver.cc:460] Webserver started at http://127.24.20.131:44405/ using document root <none> and password file <none>
I20260620 20:17:22.894109 9592 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:22.894169 9592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:22.894371 9592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:22.896329 9592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "33383f1429304a278a8c2c380c44e435"
format_stamp: "Formatted at 2026-06-20 20:17:22 on dist-test-slave-pflr"
server_key: "116686f7b5ca56a30b6ca2989927becd"
server_key_iv: "99547a31703a368cb7cd8eb9cca91c11"
server_key_version: "encryptionkey@0"
I20260620 20:17:22.896898 9592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "33383f1429304a278a8c2c380c44e435"
format_stamp: "Formatted at 2026-06-20 20:17:22 on dist-test-slave-pflr"
server_key: "116686f7b5ca56a30b6ca2989927becd"
server_key_iv: "99547a31703a368cb7cd8eb9cca91c11"
server_key_version: "encryptionkey@0"
I20260620 20:17:22.900270 9592 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:17:22.902657 9607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:22.903829 9592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:22.903914 9592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "33383f1429304a278a8c2c380c44e435"
format_stamp: "Formatted at 2026-06-20 20:17:22 on dist-test-slave-pflr"
server_key: "116686f7b5ca56a30b6ca2989927becd"
server_key_iv: "99547a31703a368cb7cd8eb9cca91c11"
server_key_version: "encryptionkey@0"
I20260620 20:17:22.904033 9592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:22.920415 9592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:22.923378 9592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:22.923568 9592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:22.924121 9592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:22.924953 9592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:22.924999 9592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:22.925065 9592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:22.925107 9592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:22.933645 9592 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:46139
I20260620 20:17:22.933670 9720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:46139 every 8 connection(s)
I20260620 20:17:22.934454 9592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:17:22.935288 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 9592
I20260620 20:17:22.935454 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:17:22.935739 24658 external_mini_cluster.cc:1428] Setting key 3b4cacdd9fe07c89214688b2b30d94e7
Jun 20 20:17:22 dist-test-slave-pflr krb5kdc[9217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986642, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:17:22.948768 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.937283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:40067 (local address 127.24.20.190:41505)
0620 20:17:22.937412 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:22.937417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:22.938333 (+ 916us) server_negotiation.cc:407] Connection header received
0620 20:17:22.939632 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:22.939651 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:22.939707 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:22.939781 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:22.940936 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.941476 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.942616 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.942835 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.942945 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.945250 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:22.945253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:22.945255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:22.945280 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:22.946865 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:22.947286 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:22.947288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:22.947289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:22.947361 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:22.947660 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:22.947663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:22.947665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:22.947844 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:22.947944 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:22.948430 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 20:17:22.948554 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:17:22.949596 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.937751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:41505 (local address 127.24.20.131:40067)
0620 20:17:22.938204 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:22.938243 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:22.939433 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:22.939946 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:22.939953 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:22.940288 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:22.940805 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:22.940820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.941601 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.941604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:22.942465 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:22.942473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.942977 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.942981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:22.943148 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.943687 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:22.943702 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:22.945088 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:22.946957 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:22.946961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:22.946970 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:22.947203 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:22.947436 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:22.947438 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:22.947440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:22.947553 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:22.947936 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:22.947941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:22.948163 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 20:17:22.948352 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 20:17:22.948666 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260620 20:17:22.950666 9721 heartbeater.cc:347] Connected to a master server at 127.24.20.190:41505
I20260620 20:17:22.950873 9721 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:22.951277 9721 heartbeater.cc:511] Master 127.24.20.190:41505 requested a full tablet report, sending...
I20260620 20:17:22.952174 9262 ts_manager.cc:194] Registered new tserver with Master: 33383f1429304a278a8c2c380c44e435 (127.24.20.131:46139)
I20260620 20:17:22.952800 9262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:40067
I20260620 20:17:22.961527 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:17:22.970835 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.963421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.24.20.190:41505)
0620 20:17:22.963569 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:22.963575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:22.963779 (+ 204us) server_negotiation.cc:407] Connection header received
0620 20:17:22.963924 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:22.963927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:22.963973 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:22.964065 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:22.964743 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.965270 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.966119 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.966335 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.966441 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.967773 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:22.967777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:22.967781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:22.967808 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:22.969120 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:22.969640 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:22.969643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:22.969645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:22.969724 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:22.969992 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:22.969996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:22.969997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:22.970211 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:22.970349 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:22.970568 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 20:17:22.970673 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:17:22.973819 9262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:17:22.976891 9262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:17:22.990458 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.984950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:46139 (local address 127.0.0.1:38712)
0620 20:17:22.985295 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:22.985482 (+ 187us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:22.985610 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:22.986162 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:22.986167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:22.986189 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:22.986342 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:22.986353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.987757 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.987760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:22.989091 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:22.989103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.989947 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.989953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:22.990059 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.990098 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:17:22.990207 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 20:17:22.990315 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":92,"threads_started":1}
I20260620 20:17:22.990478 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.985128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.24.20.131:46139)
0620 20:17:22.985452 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:22.985468 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:22.985583 (+ 115us) server_negotiation.cc:407] Connection header received
0620 20:17:22.985760 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:22.985766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:22.985983 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:22.986179 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:22.986507 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.987628 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.989223 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.989837 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.989957 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.990112 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:22.990244 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 20:17:22.990339 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":102,"threads_started":1}
I20260620 20:17:22.991254 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.985695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:34645 (local address 127.0.0.1:44726)
0620 20:17:22.985913 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:22.985925 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:22.986006 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:22.987116 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:22.987119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:22.987136 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:22.987221 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:22.987228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.988490 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.988493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:22.989743 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:22.989754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.990849 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.990853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:22.990954 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.990981 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 20:17:22.991030 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 20:17:22.991093 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":62,"thread_start_us":91,"threads_started":1}
I20260620 20:17:22.991438 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.985985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:44127 (local address 127.0.0.1:35102)
0620 20:17:22.986783 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:22.986796 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:22.986885 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:22.987283 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:22.987288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:22.987308 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:22.987384 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:22.987390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.988407 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.988412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:22.989743 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:22.989754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.990978 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.990981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:22.991077 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.991096 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 20:17:22.991215 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 20:17:22.991287 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":41,"thread_start_us":82,"threads_started":1}
I20260620 20:17:22.991472 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.986377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.24.20.130:44127)
0620 20:17:22.986758 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:22.986769 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:22.986863 (+ 94us) server_negotiation.cc:407] Connection header received
0620 20:17:22.986985 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:22.986989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:22.987156 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:22.987287 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:22.987471 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.988272 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.989887 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.990743 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.990842 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.990983 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:22.991243 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 20:17:22.991331 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":148,"threads_started":1}
I20260620 20:17:22.991915 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:22.986043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44726 (local address 127.24.20.129:34645)
0620 20:17:22.986677 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:22.986690 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:22.986714 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:17:22.986790 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:22.986796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:22.986977 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:22.987143 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:22.987342 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.988338 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.989874 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:22.990719 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:22.991479 (+ 760us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:22.991635 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:22.991684 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 20:17:22.991790 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":128,"threads_started":1}
I20260620 20:17:22.993770 9655 tablet_service.cc:1467] Processing CreateTablet for tablet b9f589884fa5474cb1dc4a9a08cfade3 (DEFAULT_TABLE table=test-table [id=94fd9a6934df4b109f7a61be38fc3325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:22.994693 9655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f589884fa5474cb1dc4a9a08cfade3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:22.995325 9387 tablet_service.cc:1467] Processing CreateTablet for tablet b9f589884fa5474cb1dc4a9a08cfade3 (DEFAULT_TABLE table=test-table [id=94fd9a6934df4b109f7a61be38fc3325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:22.995327 9521 tablet_service.cc:1467] Processing CreateTablet for tablet b9f589884fa5474cb1dc4a9a08cfade3 (DEFAULT_TABLE table=test-table [id=94fd9a6934df4b109f7a61be38fc3325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:22.996321 9521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f589884fa5474cb1dc4a9a08cfade3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:22.996321 9387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f589884fa5474cb1dc4a9a08cfade3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:23.001281 9738 tablet_bootstrap.cc:492] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435: Bootstrap starting.
I20260620 20:17:23.003087 9739 tablet_bootstrap.cc:492] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2: Bootstrap starting.
I20260620 20:17:23.003455 9738 tablet_bootstrap.cc:654] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:23.003816 9740 tablet_bootstrap.cc:492] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64: Bootstrap starting.
I20260620 20:17:23.004890 9738 log.cc:824] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:23.005324 9739 tablet_bootstrap.cc:654] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:23.005824 9740 tablet_bootstrap.cc:654] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:23.006548 9739 log.cc:824] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:23.007038 9738 tablet_bootstrap.cc:492] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435: No bootstrap required, opened a new log
I20260620 20:17:23.007287 9738 ts_tablet_manager.cc:1397] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 20:17:23.007323 9740 log.cc:824] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:23.008687 9739 tablet_bootstrap.cc:492] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2: No bootstrap required, opened a new log
I20260620 20:17:23.008857 9739 ts_tablet_manager.cc:1397] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:17:23.009208 9740 tablet_bootstrap.cc:492] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64: No bootstrap required, opened a new log
I20260620 20:17:23.009393 9740 ts_tablet_manager.cc:1397] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 20:17:23.011170 9738 raft_consensus.cc:348] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33383f1429304a278a8c2c380c44e435" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46139 } } peers { permanent_uuid: "a4ac0dabf130465e9512ee7ccdefaa64" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34645 } } peers { permanent_uuid: "2abbdd9e21104c18b51922038e6aa5f2" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44127 } }
I20260620 20:17:23.011437 9738 raft_consensus.cc:374] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:23.011518 9738 raft_consensus.cc:729] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33383f1429304a278a8c2c380c44e435, State: Initialized, Role: FOLLOWER
I20260620 20:17:23.011976 9738 consensus_queue.cc:260] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33383f1429304a278a8c2c380c44e435" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46139 } } peers { permanent_uuid: "a4ac0dabf130465e9512ee7ccdefaa64" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34645 } } peers { permanent_uuid: "2abbdd9e21104c18b51922038e6aa5f2" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44127 } }
I20260620 20:17:23.012257 9740 raft_consensus.cc:348] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33383f1429304a278a8c2c380c44e435" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46139 } } peers { permanent_uuid: "a4ac0dabf130465e9512ee7ccdefaa64" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34645 } } peers { permanent_uuid: "2abbdd9e21104c18b51922038e6aa5f2" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44127 } }
I20260620 20:17:23.012651 9721 heartbeater.cc:503] Master 127.24.20.190:41505 was elected leader, sending a full tablet report...
I20260620 20:17:23.012511 9739 raft_consensus.cc:348] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33383f1429304a278a8c2c380c44e435" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46139 } } peers { permanent_uuid: "a4ac0dabf130465e9512ee7ccdefaa64" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34645 } } peers { permanent_uuid: "2abbdd9e21104c18b51922038e6aa5f2" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44127 } }
I20260620 20:17:23.012746 9739 raft_consensus.cc:374] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:23.012809 9739 raft_consensus.cc:729] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2abbdd9e21104c18b51922038e6aa5f2, State: Initialized, Role: FOLLOWER
I20260620 20:17:23.012998 9738 ts_tablet_manager.cc:1428] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260620 20:17:23.013294 9739 consensus_queue.cc:260] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33383f1429304a278a8c2c380c44e435" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46139 } } peers { permanent_uuid: "a4ac0dabf130465e9512ee7ccdefaa64" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34645 } } peers { permanent_uuid: "2abbdd9e21104c18b51922038e6aa5f2" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44127 } }
I20260620 20:17:23.013703 9740 raft_consensus.cc:374] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:23.013818 9740 raft_consensus.cc:729] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ac0dabf130465e9512ee7ccdefaa64, State: Initialized, Role: FOLLOWER
I20260620 20:17:23.014292 9739 ts_tablet_manager.cc:1428] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:17:23.014264 9740 consensus_queue.cc:260] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33383f1429304a278a8c2c380c44e435" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46139 } } peers { permanent_uuid: "a4ac0dabf130465e9512ee7ccdefaa64" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34645 } } peers { permanent_uuid: "2abbdd9e21104c18b51922038e6aa5f2" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44127 } }
I20260620 20:17:23.014921 9453 heartbeater.cc:503] Master 127.24.20.190:41505 was elected leader, sending a full tablet report...
I20260620 20:17:23.015082 9740 ts_tablet_manager.cc:1428] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260620 20:17:23.015434 9587 heartbeater.cc:503] Master 127.24.20.190:41505 was elected leader, sending a full tablet report...
W20260620 20:17:23.025226 9454 tablet.cc:2369] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:17:23.185405 9722 tablet.cc:2369] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:17:23.187080 9745 raft_consensus.cc:482] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:23.187258 9745 raft_consensus.cc:504] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33383f1429304a278a8c2c380c44e435" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46139 } } peers { permanent_uuid: "a4ac0dabf130465e9512ee7ccdefaa64" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34645 } } peers { permanent_uuid: "2abbdd9e21104c18b51922038e6aa5f2" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44127 } }
I20260620 20:17:23.188320 9745 leader_election.cc:288] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33383f1429304a278a8c2c380c44e435 (127.24.20.131:46139), a4ac0dabf130465e9512ee7ccdefaa64 (127.24.20.129:34645)
I20260620 20:17:23.191988 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:23.188628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:46139 (local address 127.24.20.130:44783)
0620 20:17:23.188772 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:23.188790 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:23.188902 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:23.189237 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:23.189240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:23.189258 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:23.189383 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:23.189388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.190095 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.190101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:23.190939 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:23.190948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.191575 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.191579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:23.191678 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:23.191699 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:17:23.191790 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 20:17:23.191859 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 20:17:23.191994 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:23.188703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:44783 (local address 127.24.20.131:46139)
0620 20:17:23.188849 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:23.188853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:23.188871 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:17:23.189031 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:23.189038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:23.189114 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:23.189228 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:23.189486 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.189975 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.191049 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.191450 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.191577 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:23.191632 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:23.191804 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 20:17:23.191874 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:17:23.192692 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:23.188908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:34645 (local address 127.24.20.130:50613)
0620 20:17:23.189240 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:23.189252 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:23.189371 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:23.189608 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:23.189611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:23.189626 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:23.189758 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:23.189763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.190469 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.190475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:23.191437 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:23.191446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.192238 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.192241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:23.192382 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:23.192407 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:17:23.192487 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 20:17:23.192546 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":126,"threads_started":1}
I20260620 20:17:23.192725 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:23.189040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:50613 (local address 127.24.20.129:34645)
0620 20:17:23.189201 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:23.189205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:23.189334 (+ 129us) server_negotiation.cc:407] Connection header received
0620 20:17:23.189456 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:23.189459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:23.189505 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:23.189598 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:23.189864 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.190358 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.191552 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.191987 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.192126 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:23.192181 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:23.192506 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 20:17:23.192578 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 20:17:23.192677 9675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f589884fa5474cb1dc4a9a08cfade3" candidate_uuid: "2abbdd9e21104c18b51922038e6aa5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33383f1429304a278a8c2c380c44e435" is_pre_election: true
I20260620 20:17:23.193015 9675 raft_consensus.cc:2454] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2abbdd9e21104c18b51922038e6aa5f2 in term 0.
I20260620 20:17:23.193396 9407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f589884fa5474cb1dc4a9a08cfade3" candidate_uuid: "2abbdd9e21104c18b51922038e6aa5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ac0dabf130465e9512ee7ccdefaa64" is_pre_election: true
I20260620 20:17:23.193490 9476 leader_election.cc:302] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2abbdd9e21104c18b51922038e6aa5f2, 33383f1429304a278a8c2c380c44e435; no voters:
I20260620 20:17:23.193693 9407 raft_consensus.cc:2454] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2abbdd9e21104c18b51922038e6aa5f2 in term 0.
I20260620 20:17:23.193775 9745 raft_consensus.cc:2781] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:17:23.193862 9745 raft_consensus.cc:482] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:17:23.193969 9745 raft_consensus.cc:3037] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:23.195046 9745 raft_consensus.cc:504] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33383f1429304a278a8c2c380c44e435" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46139 } } peers { permanent_uuid: "a4ac0dabf130465e9512ee7ccdefaa64" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34645 } } peers { permanent_uuid: "2abbdd9e21104c18b51922038e6aa5f2" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44127 } }
I20260620 20:17:23.195386 9745 leader_election.cc:288] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [CANDIDATE]: Term 1 election: Requested vote from peers 33383f1429304a278a8c2c380c44e435 (127.24.20.131:46139), a4ac0dabf130465e9512ee7ccdefaa64 (127.24.20.129:34645)
I20260620 20:17:23.195724 9675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f589884fa5474cb1dc4a9a08cfade3" candidate_uuid: "2abbdd9e21104c18b51922038e6aa5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33383f1429304a278a8c2c380c44e435"
I20260620 20:17:23.195822 9675 raft_consensus.cc:3037] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:23.195799 9407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f589884fa5474cb1dc4a9a08cfade3" candidate_uuid: "2abbdd9e21104c18b51922038e6aa5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ac0dabf130465e9512ee7ccdefaa64"
I20260620 20:17:23.195926 9407 raft_consensus.cc:3037] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:23.197034 9675 raft_consensus.cc:2454] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2abbdd9e21104c18b51922038e6aa5f2 in term 1.
I20260620 20:17:23.197160 9407 raft_consensus.cc:2454] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2abbdd9e21104c18b51922038e6aa5f2 in term 1.
I20260620 20:17:23.197345 9476 leader_election.cc:302] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2abbdd9e21104c18b51922038e6aa5f2, 33383f1429304a278a8c2c380c44e435; no voters:
I20260620 20:17:23.197584 9745 raft_consensus.cc:2781] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:23.197795 9745 raft_consensus.cc:686] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [term 1 LEADER]: Becoming Leader. State: Replica: 2abbdd9e21104c18b51922038e6aa5f2, State: Running, Role: LEADER
I20260620 20:17:23.198146 9745 consensus_queue.cc:237] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33383f1429304a278a8c2c380c44e435" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46139 } } peers { permanent_uuid: "a4ac0dabf130465e9512ee7ccdefaa64" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34645 } } peers { permanent_uuid: "2abbdd9e21104c18b51922038e6aa5f2" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44127 } }
I20260620 20:17:23.201246 9261 catalog_manager.cc:5495] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2abbdd9e21104c18b51922038e6aa5f2 (127.24.20.130). New cstate: current_term: 1 leader_uuid: "2abbdd9e21104c18b51922038e6aa5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33383f1429304a278a8c2c380c44e435" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4ac0dabf130465e9512ee7ccdefaa64" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2abbdd9e21104c18b51922038e6aa5f2" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44127 } health_report { overall_health: HEALTHY } } }
I20260620 20:17:23.222766 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:23.219393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.24.20.130:44127)
0620 20:17:23.219512 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:23.219515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:23.219527 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:17:23.219623 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:23.219625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:23.219673 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:23.219754 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:17:23.219943 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.220412 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.221324 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.221543 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.221660 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:23.221958 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:17:23.222346 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:17:23.222490 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:23.222552 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 20:17:23.222611 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:17:23.228217 9407 raft_consensus.cc:1261] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64 [term 1 FOLLOWER]: Refusing update from remote peer 2abbdd9e21104c18b51922038e6aa5f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:23.228217 9675 raft_consensus.cc:1261] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435 [term 1 FOLLOWER]: Refusing update from remote peer 2abbdd9e21104c18b51922038e6aa5f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:23.229022 9748 consensus_queue.cc:1035] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33383f1429304a278a8c2c380c44e435" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:23.229223 9745 consensus_queue.cc:1035] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4ac0dabf130465e9512ee7ccdefaa64" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:17:23.230958 9588 tablet.cc:2369] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:17:23.237586 9752 mvcc.cc:205] Tried to move back new op lower bound from 7299017290655948800 to 7299017290542374912. Current Snapshot: MvccSnapshot[applied={T|T < 7299017290655948800}]
I20260620 20:17:23.244691 9260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:23.244824 9260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:23.247073 9260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9115992f7e38466b9ed3b96137a76a7f: Sending DeleteTablet for 3 replicas of tablet b9f589884fa5474cb1dc4a9a08cfade3
I20260620 20:17:23.247758 9655 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f589884fa5474cb1dc4a9a08cfade3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:23 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:38712
I20260620 20:17:23.247880 9521 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f589884fa5474cb1dc4a9a08cfade3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:23 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:35102
I20260620 20:17:23.248006 9387 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f589884fa5474cb1dc4a9a08cfade3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:23 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:44726
I20260620 20:17:23.248229 9761 tablet_replica.cc:330] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2: stopping tablet replica
I20260620 20:17:23.248226 9760 tablet_replica.cc:330] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435: stopping tablet replica
I20260620 20:17:23.248482 9762 tablet_replica.cc:330] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64: stopping tablet replica
I20260620 20:17:23.248490 9761 raft_consensus.cc:2229] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:17:23.248490 9760 raft_consensus.cc:2229] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:17:23.248765 9760 raft_consensus.cc:2258] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:23.248816 9761 raft_consensus.cc:2258] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:23.248864 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 9324
I20260620 20:17:23.248987 9762 raft_consensus.cc:2229] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:17:23.249303 9762 raft_consensus.cc:2258] T b9f589884fa5474cb1dc4a9a08cfade3 P a4ac0dabf130465e9512ee7ccdefaa64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:23.250769 9761 ts_tablet_manager.cc:1905] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:17:23.251260 9760 ts_tablet_manager.cc:1905] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:17:23.253466 9761 ts_tablet_manager.cc:1918] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:23.253551 9761 log.cc:1196] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/b9f589884fa5474cb1dc4a9a08cfade3
I20260620 20:17:23.253876 9761 ts_tablet_manager.cc:1939] T b9f589884fa5474cb1dc4a9a08cfade3 P 2abbdd9e21104c18b51922038e6aa5f2: Deleting consensus metadata
I20260620 20:17:23.253859 9760 ts_tablet_manager.cc:1918] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:23.253948 9760 log.cc:1196] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/b9f589884fa5474cb1dc4a9a08cfade3
I20260620 20:17:23.254269 9760 ts_tablet_manager.cc:1939] T b9f589884fa5474cb1dc4a9a08cfade3 P 33383f1429304a278a8c2c380c44e435: Deleting consensus metadata
I20260620 20:17:23.254983 9248 catalog_manager.cc:4841] TS 2abbdd9e21104c18b51922038e6aa5f2 (127.24.20.130:44127): tablet b9f589884fa5474cb1dc4a9a08cfade3 (table test-table [id=94fd9a6934df4b109f7a61be38fc3325]) successfully deleted
I20260620 20:17:23.255254 9248 catalog_manager.cc:4841] TS 33383f1429304a278a8c2c380c44e435 (127.24.20.131:46139): tablet b9f589884fa5474cb1dc4a9a08cfade3 (table test-table [id=94fd9a6934df4b109f7a61be38fc3325]) successfully deleted
W20260620 20:17:23.256644 9248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:34645 (error 108)
I20260620 20:17:23.256863 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 9458
I20260620 20:17:23.258419 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:23.258171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:34645 (local address 127.0.0.1:44740)
0620 20:17:23.258294 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:23.258359 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:34645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 20:17:23.258630 9248 catalog_manager.cc:4584] TS a4ac0dabf130465e9512ee7ccdefaa64 (127.24.20.129:34645): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9f589884fa5474cb1dc4a9a08cfade3: Network error: Client connection negotiation failed: client connection to 127.24.20.129:34645: connect: Connection refused (error 111)
I20260620 20:17:23.262888 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 9592
I20260620 20:17:23.268054 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 9230
2026-06-20T20:17:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2364 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T20:17:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:23Z Disabled control of system clock
I20260620 20:17:23.297578 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:44663
--webserver_interface=127.24.20.190
--webserver_port=0
--builtin_ntp_servers=127.24.20.148:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:44663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:17:23.404774 9767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:23.405021 9767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:23.405055 9767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:23.409062 9767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:23.409116 9767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:23.409130 9767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:23.409147 9767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:23.409162 9767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:23.414430 9767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:44663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:23.415494 9767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:23.416885 9767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:23.424918 9773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:23.424908 9772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:23.424909 9775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:23.425212 9767 server_base.cc:1034] running on GCE node
I20260620 20:17:23.425716 9767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:23.427006 9767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:23.428171 9767 hybrid_clock.cc:648] HybridClock initialized: now 1781986643428109 us; error 80 us; skew 500 ppm
I20260620 20:17:23.430322 9767 webserver.cc:460] Webserver started at http://127.24.20.190:44283/ using document root <none> and password file <none>
I20260620 20:17:23.431092 9767 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:23.431183 9767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:23.431434 9767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:23.434230 9767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "29c54a89a64c47f0a5b1026c7bc560a0"
format_stamp: "Formatted at 2026-06-20 20:17:23 on dist-test-slave-pflr"
server_key: "7923425fd80ae8db4801f2916b205c28"
server_key_iv: "761f14dd0fd9d61df0b8c77ab8a8db24"
server_key_version: "encryptionkey@0"
I20260620 20:17:23.434899 9767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "29c54a89a64c47f0a5b1026c7bc560a0"
format_stamp: "Formatted at 2026-06-20 20:17:23 on dist-test-slave-pflr"
server_key: "7923425fd80ae8db4801f2916b205c28"
server_key_iv: "761f14dd0fd9d61df0b8c77ab8a8db24"
server_key_version: "encryptionkey@0"
I20260620 20:17:23.438503 9767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260620 20:17:23.440858 9781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:23.441949 9767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:23.442090 9767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "29c54a89a64c47f0a5b1026c7bc560a0"
format_stamp: "Formatted at 2026-06-20 20:17:23 on dist-test-slave-pflr"
server_key: "7923425fd80ae8db4801f2916b205c28"
server_key_iv: "761f14dd0fd9d61df0b8c77ab8a8db24"
server_key_version: "encryptionkey@0"
I20260620 20:17:23.442194 9767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:23.463687 9767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:23.464362 9767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:23.464552 9767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:23.471273 9767 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:44663
I20260620 20:17:23.471292 9833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:44663 every 8 connection(s)
I20260620 20:17:23.472162 9767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:23.473085 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 9767
I20260620 20:17:23.473208 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:23.473527 24658 external_mini_cluster.cc:1428] Setting key 53096875f220c2f1622bd8bb410a7602
I20260620 20:17:23.475766 9834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:23.480993 9834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0: Bootstrap starting.
I20260620 20:17:23.482522 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:23.474923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.24.20.190:44663)
0620 20:17:23.475536 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:23.475550 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:23.475610 (+ 60us) server_negotiation.cc:407] Connection header received
0620 20:17:23.476290 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:23.476306 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:23.476752 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:23.477075 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:23.477476 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.478404 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.479269 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.479731 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.479889 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:23.480813 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:23.480818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:23.480829 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:23.481238 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:23.481291 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:23.481311 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:23.481414 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:23.481526 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 20:17:23.481771 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":135,"threads_started":1}
I20260620 20:17:23.483264 9834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:23.484135 9834 log.cc:824] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:23.485957 9834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0: No bootstrap required, opened a new log
I20260620 20:17:23.489321 9834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c54a89a64c47f0a5b1026c7bc560a0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44663 } }
I20260620 20:17:23.489634 9834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:23.489703 9834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c54a89a64c47f0a5b1026c7bc560a0, State: Initialized, Role: FOLLOWER
I20260620 20:17:23.490127 9834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c54a89a64c47f0a5b1026c7bc560a0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44663 } }
I20260620 20:17:23.490247 9834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:23.490291 9834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:23.490357 9834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:23.491180 9834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c54a89a64c47f0a5b1026c7bc560a0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44663 } }
I20260620 20:17:23.491451 9834 leader_election.cc:302] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29c54a89a64c47f0a5b1026c7bc560a0; no voters:
I20260620 20:17:23.491690 9834 leader_election.cc:288] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:23.491811 9839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:23.492017 9839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [term 1 LEADER]: Becoming Leader. State: Replica: 29c54a89a64c47f0a5b1026c7bc560a0, State: Running, Role: LEADER
I20260620 20:17:23.492312 9839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c54a89a64c47f0a5b1026c7bc560a0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44663 } }
I20260620 20:17:23.492931 9834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:23.494109 9840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29c54a89a64c47f0a5b1026c7bc560a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c54a89a64c47f0a5b1026c7bc560a0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44663 } } }
I20260620 20:17:23.494243 9840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:23.494498 9841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29c54a89a64c47f0a5b1026c7bc560a0. Latest consensus state: current_term: 1 leader_uuid: "29c54a89a64c47f0a5b1026c7bc560a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c54a89a64c47f0a5b1026c7bc560a0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44663 } } }
I20260620 20:17:23.494598 9841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:23.494623 9848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:23.497314 9848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:23.502673 9848 catalog_manager.cc:1347] Generated new cluster ID: 41d35d403f144ce7b52cf51f99f05e83
I20260620 20:17:23.502746 9848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:23.513115 9848 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:23.514074 9848 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:23.522667 9848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0: Generated new TSK 0
I20260620 20:17:23.523516 9848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:23.531320 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44663
--builtin_ntp_servers=127.24.20.148:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:17:23.641463 9858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:23.641691 9858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:23.641728 9858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:23.645725 9858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:23.645771 9858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:23.645874 9858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:17:23.650520 9858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:23.651484 9858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:23.652845 9858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:23.661854 9866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:23.661921 9864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:23.661931 9863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:23.662127 9858 server_base.cc:1034] running on GCE node
I20260620 20:17:23.662595 9858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:23.663165 9858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:23.664330 9858 hybrid_clock.cc:648] HybridClock initialized: now 1781986643664295 us; error 44 us; skew 500 ppm
I20260620 20:17:23.669909 9858 webserver.cc:460] Webserver started at http://127.24.20.129:39017/ using document root <none> and password file <none>
I20260620 20:17:23.670457 9858 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:23.670517 9858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:23.670759 9858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:23.672796 9858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "5033385d3c484930b5196890c43c7bb6"
format_stamp: "Formatted at 2026-06-20 20:17:23 on dist-test-slave-pflr"
server_key: "ec5155b0973ea02ef57463a7ea33f191"
server_key_iv: "1e75074dcf676afdfe9ade2b88ab02e2"
server_key_version: "encryptionkey@0"
I20260620 20:17:23.673342 9858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "5033385d3c484930b5196890c43c7bb6"
format_stamp: "Formatted at 2026-06-20 20:17:23 on dist-test-slave-pflr"
server_key: "ec5155b0973ea02ef57463a7ea33f191"
server_key_iv: "1e75074dcf676afdfe9ade2b88ab02e2"
server_key_version: "encryptionkey@0"
I20260620 20:17:23.676832 9858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:23.679032 9872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:23.680167 9858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:23.680279 9858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "5033385d3c484930b5196890c43c7bb6"
format_stamp: "Formatted at 2026-06-20 20:17:23 on dist-test-slave-pflr"
server_key: "ec5155b0973ea02ef57463a7ea33f191"
server_key_iv: "1e75074dcf676afdfe9ade2b88ab02e2"
server_key_version: "encryptionkey@0"
I20260620 20:17:23.680373 9858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:23.697624 9858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:23.698251 9858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:23.698439 9858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:23.699000 9858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:23.699882 9858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:23.699936 9858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:23.700003 9858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:23.700033 9858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:23.708894 9858 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:38303
I20260620 20:17:23.708927 9985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:38303 every 8 connection(s)
I20260620 20:17:23.709720 9858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:17:23.717716 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 9858
I20260620 20:17:23.717839 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:17:23.718199 24658 external_mini_cluster.cc:1428] Setting key c67b7f9abd148a04df5e498dc019dbbb
I20260620 20:17:23.718439 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:23.711414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:60679 (local address 127.24.20.190:44663)
0620 20:17:23.711591 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:23.711596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:23.712628 (+ 1032us) server_negotiation.cc:407] Connection header received
0620 20:17:23.713933 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:23.713936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:23.713984 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:23.714057 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:23.714904 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.715421 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.716506 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.716781 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.716862 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:23.717930 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:23.717933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:23.717935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:23.717946 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:23.717963 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:23.717971 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:23.718032 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:23.718220 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 20:17:23.718318 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:17:23.719419 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:23.711798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44663 (local address 127.24.20.129:60679)
0620 20:17:23.712446 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:23.712485 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:23.713741 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:23.714214 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:23.714219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:23.714298 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:23.714782 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:23.714794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.715522 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.715525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:23.716384 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:23.716401 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.716908 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.716911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:23.717043 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:23.717604 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:23.717605 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:23.717658 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:23.717737 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:23.718062 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:23.718071 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:23.718077 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:17:23.718207 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 20:17:23.718514 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260620 20:17:23.720227 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44663
--builtin_ntp_servers=127.24.20.148:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:17:23.720548 9986 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44663
I20260620 20:17:23.720798 9986 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:23.721725 9986 heartbeater.cc:511] Master 127.24.20.190:44663 requested a full tablet report, sending...
I20260620 20:17:23.723762 9798 ts_manager.cc:194] Registered new tserver with Master: 5033385d3c484930b5196890c43c7bb6 (127.24.20.129:38303)
I20260620 20:17:23.725713 9798 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.20.129:60679
W20260620 20:17:23.840318 9989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:23.840571 9989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:23.840626 9989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:23.844501 9989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:23.844548 9989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:23.844661 9989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:17:23.849572 9989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:23.850539 9989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:23.851891 9989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:23.861305 9997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:23.861311 9994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:23.861330 9995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:23.861624 9989 server_base.cc:1034] running on GCE node
I20260620 20:17:23.862011 9989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:23.862619 9989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:23.863801 9989 hybrid_clock.cc:648] HybridClock initialized: now 1781986643863767 us; error 44 us; skew 500 ppm
I20260620 20:17:23.865552 9989 webserver.cc:460] Webserver started at http://127.24.20.130:36531/ using document root <none> and password file <none>
I20260620 20:17:23.866101 9989 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:23.866189 9989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:23.866415 9989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:23.868410 9989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "b913ec3a554a4399a71b8d30f2244ce5"
format_stamp: "Formatted at 2026-06-20 20:17:23 on dist-test-slave-pflr"
server_key: "ac2fcde3549e8ce390c989dab76cd217"
server_key_iv: "93dd490f4b3aa6a15a302cd68a12982d"
server_key_version: "encryptionkey@0"
I20260620 20:17:23.869009 9989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "b913ec3a554a4399a71b8d30f2244ce5"
format_stamp: "Formatted at 2026-06-20 20:17:23 on dist-test-slave-pflr"
server_key: "ac2fcde3549e8ce390c989dab76cd217"
server_key_iv: "93dd490f4b3aa6a15a302cd68a12982d"
server_key_version: "encryptionkey@0"
I20260620 20:17:23.872360 9989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:23.874660 10003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:23.875780 9989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:23.875908 9989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "b913ec3a554a4399a71b8d30f2244ce5"
format_stamp: "Formatted at 2026-06-20 20:17:23 on dist-test-slave-pflr"
server_key: "ac2fcde3549e8ce390c989dab76cd217"
server_key_iv: "93dd490f4b3aa6a15a302cd68a12982d"
server_key_version: "encryptionkey@0"
I20260620 20:17:23.876015 9989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:23.894833 9989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:23.895421 9989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:23.895617 9989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:23.896147 9989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:23.897045 9989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:23.897123 9989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:23.897212 9989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:23.897264 9989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:23.905494 9989 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:45059
I20260620 20:17:23.905514 10116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:45059 every 8 connection(s)
I20260620 20:17:23.906339 9989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:17:23.907745 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 9989
I20260620 20:17:23.907860 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:17:23.908190 24658 external_mini_cluster.cc:1428] Setting key 8605e7c97eb4a6c9bae3a3f09d46f83d
I20260620 20:17:23.910297 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44663
--builtin_ntp_servers=127.24.20.148:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:17:23.915849 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:23.908181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:60099 (local address 127.24.20.190:44663)
0620 20:17:23.908322 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:23.908327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:23.909673 (+ 1346us) server_negotiation.cc:407] Connection header received
0620 20:17:23.911127 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:23.911133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:23.911195 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:23.911289 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:23.912107 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.912896 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.913952 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.914171 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.914283 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:23.915343 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:23.915347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:23.915350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:23.915366 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:23.915384 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:23.915391 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:23.915463 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:23.915652 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 20:17:23.915712 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:17:23.916673 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:23.908720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44663 (local address 127.24.20.130:60099)
0620 20:17:23.909489 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:23.909541 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:23.910929 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:23.911435 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:23.911441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:23.911521 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:23.911970 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:23.911981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.912994 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.912997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:23.913826 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:23.913834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:23.914315 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:23.914319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:23.914485 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:23.915094 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:23.915095 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:23.915132 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:23.915191 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:23.915477 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:23.915487 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:23.915495 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:17:23.915650 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 20:17:23.915903 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":147,"threads_started":1}
I20260620 20:17:23.917906 10117 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44663
I20260620 20:17:23.918128 10117 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:23.918591 10117 heartbeater.cc:511] Master 127.24.20.190:44663 requested a full tablet report, sending...
I20260620 20:17:23.919649 9798 ts_manager.cc:194] Registered new tserver with Master: b913ec3a554a4399a71b8d30f2244ce5 (127.24.20.130:45059)
I20260620 20:17:23.920508 9798 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.20.130:60099
W20260620 20:17:24.020285 10120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:24.020530 10120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:24.020568 10120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:24.025069 10120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:24.025143 10120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:24.025239 10120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:17:24.029918 10120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:24.030895 10120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:24.032261 10120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:24.041554 10126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:24.041520 10128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:24.041540 10125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:24.041831 10120 server_base.cc:1034] running on GCE node
I20260620 20:17:24.042212 10120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:24.042810 10120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:24.043975 10120 hybrid_clock.cc:648] HybridClock initialized: now 1781986644043930 us; error 58 us; skew 500 ppm
I20260620 20:17:24.045828 10120 webserver.cc:460] Webserver started at http://127.24.20.131:42295/ using document root <none> and password file <none>
I20260620 20:17:24.046375 10120 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:24.046433 10120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:24.046732 10120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:24.048731 10120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76"
format_stamp: "Formatted at 2026-06-20 20:17:24 on dist-test-slave-pflr"
server_key: "375d7c99e9785e87c24eb48224242893"
server_key_iv: "e7842eb73f8355bdd33f6ef410d0c06d"
server_key_version: "encryptionkey@0"
I20260620 20:17:24.049268 10120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76"
format_stamp: "Formatted at 2026-06-20 20:17:24 on dist-test-slave-pflr"
server_key: "375d7c99e9785e87c24eb48224242893"
server_key_iv: "e7842eb73f8355bdd33f6ef410d0c06d"
server_key_version: "encryptionkey@0"
I20260620 20:17:24.052699 10120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:17:24.054872 10134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:24.055982 10120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:24.056118 10120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76"
format_stamp: "Formatted at 2026-06-20 20:17:24 on dist-test-slave-pflr"
server_key: "375d7c99e9785e87c24eb48224242893"
server_key_iv: "e7842eb73f8355bdd33f6ef410d0c06d"
server_key_version: "encryptionkey@0"
I20260620 20:17:24.056223 10120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:24.086639 10120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:24.087337 10120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:24.087527 10120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:24.088114 10120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:24.089119 10120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:24.089193 10120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:24.089282 10120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:24.089334 10120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:24.098471 10120 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:40075
I20260620 20:17:24.098505 10247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:40075 every 8 connection(s)
I20260620 20:17:24.099324 10120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:17:24.107014 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 10120
I20260620 20:17:24.107156 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:17:24.107518 24658 external_mini_cluster.cc:1428] Setting key 1d7756b3c35274ade8649ea80e0e02b9
I20260620 20:17:24.108250 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.101120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:34081 (local address 127.24.20.190:44663)
0620 20:17:24.101282 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:24.101287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:24.102268 (+ 981us) server_negotiation.cc:407] Connection header received
0620 20:17:24.103633 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:24.103636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:24.103680 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:24.103753 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:24.104638 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.105159 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.106263 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.106461 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.106531 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.107759 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:24.107762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:24.107764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:24.107775 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:24.107792 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:24.107799 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:24.107887 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:24.108045 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 20:17:24.108111 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:17:24.109292 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.101498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44663 (local address 127.24.20.131:34081)
0620 20:17:24.102125 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:24.102165 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:24.103437 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:24.103923 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:24.103929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:24.104011 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:24.104478 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:24.104490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.105278 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.105280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:24.106159 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:24.106168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.106580 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.106582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:24.106713 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.107473 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:24.107477 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:24.107530 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:24.107614 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:24.107892 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:24.107903 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:24.107909 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:17:24.108045 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 20:17:24.108386 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":101,"threads_started":1}
I20260620 20:17:24.110343 10248 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44663
I20260620 20:17:24.110569 10248 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:24.111053 10248 heartbeater.cc:511] Master 127.24.20.190:44663 requested a full tablet report, sending...
I20260620 20:17:24.111994 9798 ts_manager.cc:194] Registered new tserver with Master: 7b217d1e78f9499d8c51ba1f5c8e1c76 (127.24.20.131:40075)
I20260620 20:17:24.112538 9798 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.20.131:34081
I20260620 20:17:24.121312 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:17:24.126511 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.123206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47996 (local address 127.24.20.190:44663)
0620 20:17:24.123318 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:24.123321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:24.123514 (+ 193us) server_negotiation.cc:407] Connection header received
0620 20:17:24.123588 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:24.123590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:24.123628 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:24.123690 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:24.123920 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.124390 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.125218 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.125417 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.125495 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.126224 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:24.126227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:24.126229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:24.126238 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:24.126252 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:24.126258 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:24.126310 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:24.126383 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 20:17:24.126421 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:17:24.129012 9798 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:17:24.131866 9798 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:17:24.145524 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.139562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:40075 (local address 127.0.0.1:43312)
0620 20:17:24.139872 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:24.139913 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:24.140021 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:24.141026 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:24.141031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:24.141054 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:24.141195 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:24.141202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.142627 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.142631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:24.143825 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:24.143834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.144933 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.144937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:24.145063 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.145102 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:17:24.145248 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 20:17:24.145358 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":22,"thread_start_us":95,"threads_started":1}
I20260620 20:17:24.145856 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.139659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:38303 (local address 127.0.0.1:49420)
0620 20:17:24.140885 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:24.140899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:24.140990 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:24.141565 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:24.141568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:24.141585 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:24.141674 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:24.141680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.142732 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.142735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:24.143960 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:24.143970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.145211 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.145218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:24.145300 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.145319 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 20:17:24.145407 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 20:17:24.145492 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"mutex_wait_us":2,"thread_start_us":104,"threads_started":1}
I20260620 20:17:24.145882 10263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.140349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.24.20.129:38303)
0620 20:17:24.141139 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:24.141150 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:24.141172 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:17:24.141252 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:24.141258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:24.141441 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:24.141599 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:24.141806 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.142624 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.144078 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.144831 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.144921 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.145100 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:24.145441 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 20:17:24.145538 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":111,"threads_started":1}
I20260620 20:17:24.145891 10260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.140282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:45059 (local address 127.0.0.1:41692)
0620 20:17:24.141177 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:24.141191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:24.141278 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:24.141754 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:24.141757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:24.141775 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:24.141853 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:24.141859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.143188 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.143191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:24.144233 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:24.144240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.145279 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.145282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:24.145359 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.145379 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:17:24.145446 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 20:17:24.145601 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":85,"thread_start_us":84,"threads_started":1}
I20260620 20:17:24.146044 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.140476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.24.20.130:45059)
0620 20:17:24.140927 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:24.140940 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:24.141300 (+ 360us) server_negotiation.cc:407] Connection header received
0620 20:17:24.141382 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:24.141389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:24.141574 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:24.141713 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:24.141971 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.143062 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.144343 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.144957 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.145054 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.145221 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:24.145514 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 20:17:24.145717 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":156,"threads_started":1}
I20260620 20:17:24.146982 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.139648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.24.20.131:40075)
0620 20:17:24.140519 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:24.140534 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:24.140556 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:17:24.140649 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:24.140656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:24.140876 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:24.141026 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:24.141344 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.142487 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.143958 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.144803 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.144921 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.145104 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:24.146746 (+ 1642us) server_negotiation.cc:299] Negotiation successful
0620 20:17:24.146839 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":117,"threads_started":1}
I20260620 20:17:24.148651 9920 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae365e93f764a9f92f578c2de7c954b (DEFAULT_TABLE table=test-table [id=22d312afa22447adb7e55e4670ae0dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:24.148743 10051 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae365e93f764a9f92f578c2de7c954b (DEFAULT_TABLE table=test-table [id=22d312afa22447adb7e55e4670ae0dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:24.149601 9920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae365e93f764a9f92f578c2de7c954b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:24.149276 10182 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae365e93f764a9f92f578c2de7c954b (DEFAULT_TABLE table=test-table [id=22d312afa22447adb7e55e4670ae0dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:24.149730 10051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae365e93f764a9f92f578c2de7c954b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:24.150132 10182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae365e93f764a9f92f578c2de7c954b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:24.155246 10265 tablet_bootstrap.cc:492] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6: Bootstrap starting.
I20260620 20:17:24.157060 10266 tablet_bootstrap.cc:492] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76: Bootstrap starting.
I20260620 20:17:24.157598 10265 tablet_bootstrap.cc:654] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:24.157640 10267 tablet_bootstrap.cc:492] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5: Bootstrap starting.
I20260620 20:17:24.158317 10265 log.cc:824] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:24.159147 10266 tablet_bootstrap.cc:654] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:24.159559 10267 tablet_bootstrap.cc:654] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:24.159957 10266 log.cc:824] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:24.160169 10265 tablet_bootstrap.cc:492] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6: No bootstrap required, opened a new log
I20260620 20:17:24.160351 10265 ts_tablet_manager.cc:1397] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:17:24.160363 10267 log.cc:824] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:24.161851 10266 tablet_bootstrap.cc:492] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76: No bootstrap required, opened a new log
I20260620 20:17:24.162058 10266 ts_tablet_manager.cc:1397] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:17:24.162485 10267 tablet_bootstrap.cc:492] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5: No bootstrap required, opened a new log
I20260620 20:17:24.162672 10267 ts_tablet_manager.cc:1397] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:17:24.163854 10265 raft_consensus.cc:348] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 40075 } } peers { permanent_uuid: "5033385d3c484930b5196890c43c7bb6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38303 } } peers { permanent_uuid: "b913ec3a554a4399a71b8d30f2244ce5" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45059 } }
I20260620 20:17:24.164091 10265 raft_consensus.cc:374] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:24.164143 10265 raft_consensus.cc:729] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5033385d3c484930b5196890c43c7bb6, State: Initialized, Role: FOLLOWER
I20260620 20:17:24.164572 10265 consensus_queue.cc:260] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 40075 } } peers { permanent_uuid: "5033385d3c484930b5196890c43c7bb6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38303 } } peers { permanent_uuid: "b913ec3a554a4399a71b8d30f2244ce5" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45059 } }
I20260620 20:17:24.165249 9986 heartbeater.cc:503] Master 127.24.20.190:44663 was elected leader, sending a full tablet report...
I20260620 20:17:24.165542 10265 ts_tablet_manager.cc:1428] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260620 20:17:24.166340 10266 raft_consensus.cc:348] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 40075 } } peers { permanent_uuid: "5033385d3c484930b5196890c43c7bb6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38303 } } peers { permanent_uuid: "b913ec3a554a4399a71b8d30f2244ce5" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45059 } }
I20260620 20:17:24.166590 10266 raft_consensus.cc:374] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:24.166630 10266 raft_consensus.cc:729] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b217d1e78f9499d8c51ba1f5c8e1c76, State: Initialized, Role: FOLLOWER
I20260620 20:17:24.166981 10266 consensus_queue.cc:260] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 40075 } } peers { permanent_uuid: "5033385d3c484930b5196890c43c7bb6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38303 } } peers { permanent_uuid: "b913ec3a554a4399a71b8d30f2244ce5" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45059 } }
I20260620 20:17:24.167289 10267 raft_consensus.cc:348] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 40075 } } peers { permanent_uuid: "5033385d3c484930b5196890c43c7bb6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38303 } } peers { permanent_uuid: "b913ec3a554a4399a71b8d30f2244ce5" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45059 } }
I20260620 20:17:24.167505 10248 heartbeater.cc:503] Master 127.24.20.190:44663 was elected leader, sending a full tablet report...
I20260620 20:17:24.167528 10267 raft_consensus.cc:374] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:24.167574 10267 raft_consensus.cc:729] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b913ec3a554a4399a71b8d30f2244ce5, State: Initialized, Role: FOLLOWER
I20260620 20:17:24.167816 10266 ts_tablet_manager.cc:1428] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:17:24.167958 10267 consensus_queue.cc:260] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 40075 } } peers { permanent_uuid: "5033385d3c484930b5196890c43c7bb6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38303 } } peers { permanent_uuid: "b913ec3a554a4399a71b8d30f2244ce5" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45059 } }
I20260620 20:17:24.168973 10117 heartbeater.cc:503] Master 127.24.20.190:44663 was elected leader, sending a full tablet report...
I20260620 20:17:24.169716 10267 ts_tablet_manager.cc:1428] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 20:17:24.178193 10272 raft_consensus.cc:482] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:24.178319 10272 raft_consensus.cc:504] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 40075 } } peers { permanent_uuid: "5033385d3c484930b5196890c43c7bb6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38303 } } peers { permanent_uuid: "b913ec3a554a4399a71b8d30f2244ce5" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45059 } }
I20260620 20:17:24.179477 10272 leader_election.cc:288] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5033385d3c484930b5196890c43c7bb6 (127.24.20.129:38303), b913ec3a554a4399a71b8d30f2244ce5 (127.24.20.130:45059)
I20260620 20:17:24.183208 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.179650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:38303 (local address 127.24.20.131:39659)
0620 20:17:24.179808 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:24.179823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:24.179907 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:24.180171 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:24.180175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:24.180193 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:24.180314 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:24.180322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.181135 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.181138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:24.181950 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:24.181958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.182736 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.182743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:24.182844 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.182908 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 20:17:24.182975 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 20:17:24.183074 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":20}
I20260620 20:17:24.183207 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.179831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:39887 (local address 127.24.20.130:45059)
0620 20:17:24.179997 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:24.180002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:24.180053 (+ 51us) server_negotiation.cc:407] Connection header received
0620 20:17:24.180196 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:24.180199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:24.180240 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:24.180317 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:24.180670 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.181186 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.182189 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.182691 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.182801 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.182848 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:24.183035 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 20:17:24.183099 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:17:24.183207 10263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.179699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:39659 (local address 127.24.20.129:38303)
0620 20:17:24.179807 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:24.179812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:24.179941 (+ 129us) server_negotiation.cc:407] Connection header received
0620 20:17:24.180001 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:24.180005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:24.180060 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:24.180136 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:24.180420 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.181034 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.182068 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.182525 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.182624 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.182660 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:24.183017 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 20:17:24.183087 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 20:17:24.183243 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.179685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:45059 (local address 127.24.20.131:39887)
0620 20:17:24.179981 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:24.179994 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:24.180100 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:24.180338 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:24.180340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:24.180353 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:24.180512 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:24.180518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.181289 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.181292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:24.182055 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:24.182065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.182805 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.182807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:24.182905 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.182922 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 20:17:24.183012 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 20:17:24.183087 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":5,"thread_start_us":95,"threads_started":1}
I20260620 20:17:24.183871 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae365e93f764a9f92f578c2de7c954b" candidate_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5033385d3c484930b5196890c43c7bb6" is_pre_election: true
I20260620 20:17:24.183902 10071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae365e93f764a9f92f578c2de7c954b" candidate_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b913ec3a554a4399a71b8d30f2244ce5" is_pre_election: true
I20260620 20:17:24.184155 10071 raft_consensus.cc:2454] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b217d1e78f9499d8c51ba1f5c8e1c76 in term 0.
I20260620 20:17:24.184166 9940 raft_consensus.cc:2454] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b217d1e78f9499d8c51ba1f5c8e1c76 in term 0.
I20260620 20:17:24.184661 10135 leader_election.cc:302] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b217d1e78f9499d8c51ba1f5c8e1c76, b913ec3a554a4399a71b8d30f2244ce5; no voters:
I20260620 20:17:24.184932 10272 raft_consensus.cc:2781] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:17:24.185011 10272 raft_consensus.cc:482] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:17:24.185062 10272 raft_consensus.cc:3037] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:24.186009 10272 raft_consensus.cc:504] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 40075 } } peers { permanent_uuid: "5033385d3c484930b5196890c43c7bb6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38303 } } peers { permanent_uuid: "b913ec3a554a4399a71b8d30f2244ce5" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45059 } }
I20260620 20:17:24.186365 10272 leader_election.cc:288] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [CANDIDATE]: Term 1 election: Requested vote from peers 5033385d3c484930b5196890c43c7bb6 (127.24.20.129:38303), b913ec3a554a4399a71b8d30f2244ce5 (127.24.20.130:45059)
I20260620 20:17:24.186611 9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae365e93f764a9f92f578c2de7c954b" candidate_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5033385d3c484930b5196890c43c7bb6"
I20260620 20:17:24.186685 10071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae365e93f764a9f92f578c2de7c954b" candidate_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b913ec3a554a4399a71b8d30f2244ce5"
I20260620 20:17:24.186762 9940 raft_consensus.cc:3037] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:24.186820 10071 raft_consensus.cc:3037] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:24.187721 9940 raft_consensus.cc:2454] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b217d1e78f9499d8c51ba1f5c8e1c76 in term 1.
I20260620 20:17:24.187784 10071 raft_consensus.cc:2454] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b217d1e78f9499d8c51ba1f5c8e1c76 in term 1.
I20260620 20:17:24.188021 10136 leader_election.cc:302] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5033385d3c484930b5196890c43c7bb6, 7b217d1e78f9499d8c51ba1f5c8e1c76; no voters:
I20260620 20:17:24.188259 10272 raft_consensus.cc:2781] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:24.188476 10272 raft_consensus.cc:686] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [term 1 LEADER]: Becoming Leader. State: Replica: 7b217d1e78f9499d8c51ba1f5c8e1c76, State: Running, Role: LEADER
I20260620 20:17:24.188832 10272 consensus_queue.cc:237] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 40075 } } peers { permanent_uuid: "5033385d3c484930b5196890c43c7bb6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38303 } } peers { permanent_uuid: "b913ec3a554a4399a71b8d30f2244ce5" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45059 } }
I20260620 20:17:24.191632 9797 catalog_manager.cc:5495] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b217d1e78f9499d8c51ba1f5c8e1c76 (127.24.20.131). New cstate: current_term: 1 leader_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b217d1e78f9499d8c51ba1f5c8e1c76" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 40075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5033385d3c484930b5196890c43c7bb6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b913ec3a554a4399a71b8d30f2244ce5" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45059 } health_report { overall_health: UNKNOWN } } }
I20260620 20:17:24.203944 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.200478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.24.20.131:40075)
0620 20:17:24.200620 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:24.200626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:24.200678 (+ 52us) server_negotiation.cc:407] Connection header received
0620 20:17:24.200848 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:24.200851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:24.200899 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:24.201002 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:17:24.201243 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.201728 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.202568 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:24.202783 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:24.202891 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:24.203187 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:17:24.203567 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:17:24.203701 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:24.203741 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 20:17:24.203787 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:17:24.209213 10071 raft_consensus.cc:1261] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5 [term 1 FOLLOWER]: Refusing update from remote peer 7b217d1e78f9499d8c51ba1f5c8e1c76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:24.209214 9940 raft_consensus.cc:1261] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6 [term 1 FOLLOWER]: Refusing update from remote peer 7b217d1e78f9499d8c51ba1f5c8e1c76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:24.209842 10275 consensus_queue.cc:1035] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5033385d3c484930b5196890c43c7bb6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 38303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:24.209997 10272 consensus_queue.cc:1035] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b913ec3a554a4399a71b8d30f2244ce5" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:17:24.210796 9987 tablet.cc:2369] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:17:24.218156 10280 mvcc.cc:205] Tried to move back new op lower bound from 7299017294674575360 to 7299017294599815168. Current Snapshot: MvccSnapshot[applied={T|T < 7299017294674575360}]
I20260620 20:17:24.225598 9798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:24.225759 9798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:24.227973 9798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29c54a89a64c47f0a5b1026c7bc560a0: Sending DeleteTablet for 3 replicas of tablet 7ae365e93f764a9f92f578c2de7c954b
I20260620 20:17:24.228693 9920 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae365e93f764a9f92f578c2de7c954b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:24 UTC) from {username='slave'} at 127.0.0.1:49420
I20260620 20:17:24.228704 10182 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae365e93f764a9f92f578c2de7c954b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:24 UTC) from {username='slave'} at 127.0.0.1:43312
I20260620 20:17:24.228813 10051 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae365e93f764a9f92f578c2de7c954b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:24 UTC) from {username='slave'} at 127.0.0.1:41692
I20260620 20:17:24.229086 10288 tablet_replica.cc:330] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6: stopping tablet replica
I20260620 20:17:24.229372 10288 raft_consensus.cc:2229] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:17:24.229619 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 9858
I20260620 20:17:24.229679 10288 raft_consensus.cc:2258] T 7ae365e93f764a9f92f578c2de7c954b P 5033385d3c484930b5196890c43c7bb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:24.233456 10290 tablet_replica.cc:330] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5: stopping tablet replica
I20260620 20:17:24.233819 10290 raft_consensus.cc:2229] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:17:24.233861 10289 tablet_replica.cc:330] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76: stopping tablet replica
I20260620 20:17:24.234143 10289 raft_consensus.cc:2229] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:17:24.234179 10290 raft_consensus.cc:2258] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:24.234526 10289 raft_consensus.cc:2258] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:24.236083 10289 ts_tablet_manager.cc:1905] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 20:17:24.236548 9783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:38303 (error 108)
I20260620 20:17:24.237613 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.237258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:38303 (local address 127.0.0.1:49428)
0620 20:17:24.237428 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:24.237492 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:38303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 20:17:24.237655 10290 ts_tablet_manager.cc:1905] T 7ae365e93f764a9f92f578c2de7c954b P b913ec3a554a4399a71b8d30f2244ce5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 20:17:24.237778 9783 catalog_manager.cc:4584] TS 5033385d3c484930b5196890c43c7bb6 (127.24.20.129:38303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ae365e93f764a9f92f578c2de7c954b: Network error: Client connection negotiation failed: client connection to 127.24.20.129:38303: connect: Connection refused (error 111)
I20260620 20:17:24.238154 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 9989
I20260620 20:17:24.239138 10289 ts_tablet_manager.cc:1918] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:24.239234 10289 log.cc:1196] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/7ae365e93f764a9f92f578c2de7c954b
I20260620 20:17:24.239545 10289 ts_tablet_manager.cc:1939] T 7ae365e93f764a9f92f578c2de7c954b P 7b217d1e78f9499d8c51ba1f5c8e1c76: Deleting consensus metadata
I20260620 20:17:24.240700 9784 catalog_manager.cc:4841] TS 7b217d1e78f9499d8c51ba1f5c8e1c76 (127.24.20.131:40075): tablet 7ae365e93f764a9f92f578c2de7c954b (table test-table [id=22d312afa22447adb7e55e4670ae0dbc]) successfully deleted
I20260620 20:17:24.244834 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 10120
I20260620 20:17:24.245020 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:24.244763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:45059 (local address 127.0.0.1:41696)
0620 20:17:24.244887 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:24.244946 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.130:45059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 20:17:24.250164 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 9767
2026-06-20T20:17:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (981 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:24 dist-test-slave-pflr krb5kdc[10294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:24 dist-test-slave-pflr krb5kdc[10294](info): set up 2 sockets
Jun 20 20:17:24 dist-test-slave-pflr krb5kdc[10294](info): commencing operation
krb5kdc: starting...
W20260620 20:17:25.300860 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:25 dist-test-slave-pflr krb5kdc[10294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:17:25.419313 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:44237
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:44237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:25.529065 10307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:25.529353 10307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:25.529426 10307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:25.533420 10307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:25.533499 10307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:25.533536 10307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:25.533594 10307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:25.533634 10307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:25.538736 10307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:44237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:44237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:25.539748 10307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:25.540818 10307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:25.549243 10315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:25.549286 10313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:25.549347 10307 server_base.cc:1034] running on GCE node
W20260620 20:17:25.549659 10312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:25.550177 10307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:25.551177 10307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:25.552363 10307 hybrid_clock.cc:648] HybridClock initialized: now 1781986645552349 us; error 30 us; skew 500 ppm
Jun 20 20:17:25 dist-test-slave-pflr krb5kdc[10294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986645, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:25.555083 10307 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:17:25.555935 10307 webserver.cc:460] Webserver started at http://127.24.20.190:46539/ using document root <none> and password file <none>
I20260620 20:17:25.556471 10307 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:25.556562 10307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:25.556803 10307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:25.558782 10307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "4cca064fc502459f850f6e7beef07a31"
format_stamp: "Formatted at 2026-06-20 20:17:25 on dist-test-slave-pflr"
server_key: "42e6c3a2ab7b9e7b1d054ede9bc5c688"
server_key_iv: "4307ff5c89a95233e73fbc0bf0659cd3"
server_key_version: "encryptionkey@0"
I20260620 20:17:25.559325 10307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "4cca064fc502459f850f6e7beef07a31"
format_stamp: "Formatted at 2026-06-20 20:17:25 on dist-test-slave-pflr"
server_key: "42e6c3a2ab7b9e7b1d054ede9bc5c688"
server_key_iv: "4307ff5c89a95233e73fbc0bf0659cd3"
server_key_version: "encryptionkey@0"
I20260620 20:17:25.563045 10307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 20:17:25.565438 10322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:25.566668 10307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:17:25.566798 10307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "4cca064fc502459f850f6e7beef07a31"
format_stamp: "Formatted at 2026-06-20 20:17:25 on dist-test-slave-pflr"
server_key: "42e6c3a2ab7b9e7b1d054ede9bc5c688"
server_key_iv: "4307ff5c89a95233e73fbc0bf0659cd3"
server_key_version: "encryptionkey@0"
I20260620 20:17:25.566905 10307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:25.582319 10307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:25.585304 10307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:25.585490 10307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:25.592358 10307 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:44237
I20260620 20:17:25.592386 10374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:44237 every 8 connection(s)
I20260620 20:17:25.593253 10307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:25.595110 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 10307
I20260620 20:17:25.595266 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:25.595623 24658 external_mini_cluster.cc:1428] Setting key 68cce9888151b451372f64f4b1efeca2
I20260620 20:17:25.596361 10375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:25.602265 10375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31: Bootstrap starting.
Jun 20 20:17:25 dist-test-slave-pflr krb5kdc[10294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:17:25.604749 10375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:25.605701 10375 log.cc:824] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:25.607659 10375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31: No bootstrap required, opened a new log
I20260620 20:17:25.609318 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:25.597109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.24.20.190:44237)
0620 20:17:25.597547 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:25.597563 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:25.597599 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:17:25.598216 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:25.598232 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:25.598531 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:25.598850 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:25.599630 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:25.600638 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:25.601494 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:25.601957 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:25.602130 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:25.604536 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:25.604541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:25.604554 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:25.604582 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:25.606581 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:25.607023 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:25.607029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:25.607035 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:25.607152 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:25.607428 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:25.607432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:25.607434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:25.607849 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:25.608062 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:25.608293 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 20:17:25.608578 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":154,"threads_started":1}
I20260620 20:17:25.611657 10375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cca064fc502459f850f6e7beef07a31" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44237 } }
I20260620 20:17:25.611897 10375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:25.611977 10375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cca064fc502459f850f6e7beef07a31, State: Initialized, Role: FOLLOWER
I20260620 20:17:25.612385 10375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cca064fc502459f850f6e7beef07a31" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44237 } }
I20260620 20:17:25.612532 10375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:25.612632 10375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:25.612753 10375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:25.613716 10375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cca064fc502459f850f6e7beef07a31" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44237 } }
I20260620 20:17:25.614049 10375 leader_election.cc:302] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cca064fc502459f850f6e7beef07a31; no voters:
I20260620 20:17:25.614336 10375 leader_election.cc:288] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:25.614467 10380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:25.614696 10380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [term 1 LEADER]: Becoming Leader. State: Replica: 4cca064fc502459f850f6e7beef07a31, State: Running, Role: LEADER
I20260620 20:17:25.614981 10380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cca064fc502459f850f6e7beef07a31" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44237 } }
I20260620 20:17:25.615530 10375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:25.616577 10381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cca064fc502459f850f6e7beef07a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cca064fc502459f850f6e7beef07a31" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44237 } } }
I20260620 20:17:25.616726 10381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:25.617018 10382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cca064fc502459f850f6e7beef07a31. Latest consensus state: current_term: 1 leader_uuid: "4cca064fc502459f850f6e7beef07a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cca064fc502459f850f6e7beef07a31" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 44237 } } }
I20260620 20:17:25.617115 10382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:25.617202 10389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:25.619915 10389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:25.625208 10389 catalog_manager.cc:1347] Generated new cluster ID: 2e9831325e07461796b3f86dab45f876
I20260620 20:17:25.625269 10389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:25.637279 10389 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:25.638491 10389 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:25.645730 10389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31: Generated new TSK 0
I20260620 20:17:25.646486 10389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:17:25.675109 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44237
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:25.782441 10401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:25.782686 10401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:25.782754 10401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:25.786883 10401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:25.786940 10401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:25.787036 10401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:17:25.792487 10401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:25.793522 10401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:25.794513 10401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:25.803968 10406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:25.804003 10401 server_base.cc:1034] running on GCE node
W20260620 20:17:25.804039 10407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:25.804178 10409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:25.804525 10401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:25.805192 10401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:25.806351 10401 hybrid_clock.cc:648] HybridClock initialized: now 1781986645806339 us; error 27 us; skew 500 ppm
Jun 20 20:17:25 dist-test-slave-pflr krb5kdc[10294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986645, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:25.809128 10401 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:17:25.809947 10401 webserver.cc:460] Webserver started at http://127.24.20.129:36607/ using document root <none> and password file <none>
I20260620 20:17:25.810492 10401 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:25.810595 10401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:25.810813 10401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:25.812924 10401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "09764f01deb041ebbf05bbf53fe5a3a6"
format_stamp: "Formatted at 2026-06-20 20:17:25 on dist-test-slave-pflr"
server_key: "e8f1a962f2d089de5d53cda6163a1e2d"
server_key_iv: "c6a2f0a809431a06884e01ddc7508051"
server_key_version: "encryptionkey@0"
I20260620 20:17:25.813467 10401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "09764f01deb041ebbf05bbf53fe5a3a6"
format_stamp: "Formatted at 2026-06-20 20:17:25 on dist-test-slave-pflr"
server_key: "e8f1a962f2d089de5d53cda6163a1e2d"
server_key_iv: "c6a2f0a809431a06884e01ddc7508051"
server_key_version: "encryptionkey@0"
I20260620 20:17:25.817269 10401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 20:17:25.819530 10416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:25.820685 10401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260620 20:17:25.820806 10401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "09764f01deb041ebbf05bbf53fe5a3a6"
format_stamp: "Formatted at 2026-06-20 20:17:25 on dist-test-slave-pflr"
server_key: "e8f1a962f2d089de5d53cda6163a1e2d"
server_key_iv: "c6a2f0a809431a06884e01ddc7508051"
server_key_version: "encryptionkey@0"
I20260620 20:17:25.820906 10401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:25.845413 10401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:25.848805 10401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:25.849007 10401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:25.849601 10401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:25.850399 10401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:25.850468 10401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:25.850538 10401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:25.850594 10401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:25.859587 10401 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:41649
I20260620 20:17:25.859607 10529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:41649 every 8 connection(s)
I20260620 20:17:25.860478 10401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:17:25.861173 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 10401
I20260620 20:17:25.861317 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:17:25.861618 24658 external_mini_cluster.cc:1428] Setting key c2db8348d8faa3f47779e78c3c103407
Jun 20 20:17:25 dist-test-slave-pflr krb5kdc[10294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986645, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
I20260620 20:17:25.874670 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:25.862399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:34027 (local address 127.24.20.190:44237)
0620 20:17:25.862534 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:25.862538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:25.863492 (+ 954us) server_negotiation.cc:407] Connection header received
0620 20:17:25.865329 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:25.865333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:25.865383 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:25.865467 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:25.866583 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:25.867115 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:25.868234 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:25.868473 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:25.868587 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:25.871338 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:25.871341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:25.871343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:25.871367 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:25.872585 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:25.873116 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:25.873119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:25.873120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:25.873215 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:25.873618 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:25.873621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:25.873622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:25.873785 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:25.873919 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:25.874371 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 20:17:25.874497 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:17:25.875531 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:25.862797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44237 (local address 127.24.20.129:34027)
0620 20:17:25.863324 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:25.863378 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:25.865135 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:25.865654 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:25.865660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:25.865990 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:25.866443 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:25.866456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:25.867239 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:25.867243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:25.868110 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:25.868119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:25.868648 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:25.868652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:25.868815 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:25.869432 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:25.869447 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:25.871161 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:25.872769 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:25.872773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:25.872784 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:25.873023 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:25.873327 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:25.873332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:25.873336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:25.873516 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:25.873895 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:25.873901 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:25.874147 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 20:17:25.874330 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 20:17:25.874636 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":139,"threads_started":1}
I20260620 20:17:25.876963 10530 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44237
I20260620 20:17:25.877198 10530 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:25.877657 10530 heartbeater.cc:511] Master 127.24.20.190:44237 requested a full tablet report, sending...
I20260620 20:17:25.879312 10339 ts_manager.cc:194] Registered new tserver with Master: 09764f01deb041ebbf05bbf53fe5a3a6 (127.24.20.129:41649)
I20260620 20:17:25.881009 10339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:34027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:17:25.889302 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44237
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:25.994637 10535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:25.994874 10535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:25.994913 10535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:25.998799 10535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:25.998847 10535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:25.998947 10535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:17:26.003990 10535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:26.005003 10535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:26.006006 10535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:26.015295 10540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:26.015342 10541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:26.015359 10543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:26.015630 10535 server_base.cc:1034] running on GCE node
I20260620 20:17:26.016072 10535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:26.016866 10535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:26.018075 10535 hybrid_clock.cc:648] HybridClock initialized: now 1781986646018060 us; error 27 us; skew 500 ppm
Jun 20 20:17:26 dist-test-slave-pflr krb5kdc[10294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986646, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:26.024747 10535 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:17:26.025707 10535 webserver.cc:460] Webserver started at http://127.24.20.130:44775/ using document root <none> and password file <none>
I20260620 20:17:26.026268 10535 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:26.026360 10535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:26.026572 10535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:26.028544 10535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "0dd4656f3f49423ebc2a86d610a12dad"
format_stamp: "Formatted at 2026-06-20 20:17:26 on dist-test-slave-pflr"
server_key: "352a8dd851ea394b471d11c8a6f6cd2a"
server_key_iv: "630a34f7d9c8150cee139a3a97309ee3"
server_key_version: "encryptionkey@0"
I20260620 20:17:26.029167 10535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "0dd4656f3f49423ebc2a86d610a12dad"
format_stamp: "Formatted at 2026-06-20 20:17:26 on dist-test-slave-pflr"
server_key: "352a8dd851ea394b471d11c8a6f6cd2a"
server_key_iv: "630a34f7d9c8150cee139a3a97309ee3"
server_key_version: "encryptionkey@0"
I20260620 20:17:26.032779 10535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:26.035045 10550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:26.036152 10535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:26.036286 10535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "0dd4656f3f49423ebc2a86d610a12dad"
format_stamp: "Formatted at 2026-06-20 20:17:26 on dist-test-slave-pflr"
server_key: "352a8dd851ea394b471d11c8a6f6cd2a"
server_key_iv: "630a34f7d9c8150cee139a3a97309ee3"
server_key_version: "encryptionkey@0"
I20260620 20:17:26.036394 10535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:26.065733 10535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:26.068679 10535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:26.068889 10535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:26.069545 10535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:26.070422 10535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:26.070498 10535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:26.070570 10535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:26.070614 10535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:26.079602 10535 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:34055
I20260620 20:17:26.079638 10663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:34055 every 8 connection(s)
I20260620 20:17:26.080502 10535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:17:26.085016 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 10535
I20260620 20:17:26.085177 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:17:26.085490 24658 external_mini_cluster.cc:1428] Setting key 1f00a7f27bc013616d373be28cdce700
Jun 20 20:17:26 dist-test-slave-pflr krb5kdc[10294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986646, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
I20260620 20:17:26.094496 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.082306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:36397 (local address 127.24.20.190:44237)
0620 20:17:26.082492 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:26.082496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:26.083200 (+ 704us) server_negotiation.cc:407] Connection header received
0620 20:17:26.084742 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:26.084746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:26.084810 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:26.084914 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:26.086282 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.086810 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.088084 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.088309 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.088436 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.090950 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:26.090953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:26.090955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:26.090976 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:26.092498 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:26.093016 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:26.093021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:26.093023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:26.093103 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:26.093440 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:26.093445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:26.093447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:26.093617 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:26.093725 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:26.094209 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 20:17:26.094329 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 20:17:26.095252 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.082594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44237 (local address 127.24.20.130:36397)
0620 20:17:26.083030 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:26.083084 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:26.084371 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:26.085068 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:26.085076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:26.085630 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:26.086130 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.086144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.086945 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.086949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.087957 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.087967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.088477 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.088482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.088707 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.089336 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:26.089399 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:26.090776 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:26.092639 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:26.092644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:26.092658 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:26.092911 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:26.093197 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:26.093201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:26.093203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:26.093337 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:26.093722 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:26.093729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:26.093970 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 20:17:26.094152 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 20:17:26.094470 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260620 20:17:26.096370 10664 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44237
I20260620 20:17:26.096701 10664 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
I20260620 20:17:26.097259 10664 heartbeater.cc:511] Master 127.24.20.190:44237 requested a full tablet report, sending...
I20260620 20:17:26.098325 10339 ts_manager.cc:194] Registered new tserver with Master: 0dd4656f3f49423ebc2a86d610a12dad (127.24.20.130:34055)
I20260620 20:17:26.098887 10339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:36397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:17:26.112277 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44237
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:26.220759 10669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:26.220995 10669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:26.221072 10669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:26.225072 10669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:26.225122 10669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:26.225224 10669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:17:26.230291 10669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:44237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:26.231269 10669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:26.232240 10669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:26.241302 10677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:26.241333 10674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:26.241302 10675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:26.241498 10669 server_base.cc:1034] running on GCE node
I20260620 20:17:26.241931 10669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:26.242560 10669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:26.243741 10669 hybrid_clock.cc:648] HybridClock initialized: now 1781986646243725 us; error 27 us; skew 500 ppm
Jun 20 20:17:26 dist-test-slave-pflr krb5kdc[10294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986646, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:26.246308 10669 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:17:26.247084 10669 webserver.cc:460] Webserver started at http://127.24.20.131:42277/ using document root <none> and password file <none>
I20260620 20:17:26.247654 10669 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:26.247766 10669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:26.247985 10669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:26.249933 10669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "2e092b59fbee4c37933513e82fc48c63"
format_stamp: "Formatted at 2026-06-20 20:17:26 on dist-test-slave-pflr"
server_key: "057f9978f337296577d2211df418e5d6"
server_key_iv: "394427d825010f7dc2b10cf962f3d3d8"
server_key_version: "encryptionkey@0"
I20260620 20:17:26.250475 10669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "2e092b59fbee4c37933513e82fc48c63"
format_stamp: "Formatted at 2026-06-20 20:17:26 on dist-test-slave-pflr"
server_key: "057f9978f337296577d2211df418e5d6"
server_key_iv: "394427d825010f7dc2b10cf962f3d3d8"
server_key_version: "encryptionkey@0"
I20260620 20:17:26.253902 10669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:26.256166 10684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:26.257352 10669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:26.257485 10669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "2e092b59fbee4c37933513e82fc48c63"
format_stamp: "Formatted at 2026-06-20 20:17:26 on dist-test-slave-pflr"
server_key: "057f9978f337296577d2211df418e5d6"
server_key_iv: "394427d825010f7dc2b10cf962f3d3d8"
server_key_version: "encryptionkey@0"
I20260620 20:17:26.257589 10669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:26.279172 10669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:26.281858 10669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:26.282078 10669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:26.282662 10669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:26.283447 10669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:26.283515 10669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:26.283581 10669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:26.283617 10669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:26.292573 10669 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:41323
I20260620 20:17:26.292621 10797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:41323 every 8 connection(s)
I20260620 20:17:26.293418 10669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:17:26.298276 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 10669
I20260620 20:17:26.298434 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:17:26.298710 24658 external_mini_cluster.cc:1428] Setting key 2f55b352d91d034f5df80b37de32cffc
Jun 20 20:17:26 dist-test-slave-pflr krb5kdc[10294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986646, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:17:26.307330 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.295100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:35791 (local address 127.24.20.190:44237)
0620 20:17:26.295244 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:26.295248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:26.296008 (+ 760us) server_negotiation.cc:407] Connection header received
0620 20:17:26.297337 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:26.297340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:26.297400 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:26.297509 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:26.298931 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.299705 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.300918 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.301212 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.301296 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.303873 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:26.303876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:26.303878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:26.303899 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:26.305466 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:26.305972 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:26.305975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:26.305976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:26.306055 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:26.306367 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:26.306369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:26.306370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:26.306557 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:26.306636 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:26.307105 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 20:17:26.307201 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:17:26.308108 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.295445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:44237 (local address 127.24.20.131:35791)
0620 20:17:26.295867 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:26.295907 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:26.297134 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:26.297784 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:26.297792 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:26.298144 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:26.298710 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.298724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.299812 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.299815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.300797 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.300809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.301357 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.301361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.301515 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.302234 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:26.302250 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:26.303698 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:26.305583 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:26.305589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:26.305600 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:26.305850 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:26.306153 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:26.306155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:26.306156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:26.306279 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:26.306644 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:26.306649 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:26.306869 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 20:17:26.307021 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 20:17:26.307280 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20260620 20:17:26.309386 10798 heartbeater.cc:347] Connected to a master server at 127.24.20.190:44237
I20260620 20:17:26.309624 10798 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:26.310114 10798 heartbeater.cc:511] Master 127.24.20.190:44237 requested a full tablet report, sending...
I20260620 20:17:26.311162 10339 ts_manager.cc:194] Registered new tserver with Master: 2e092b59fbee4c37933513e82fc48c63 (127.24.20.131:41323)
I20260620 20:17:26.311911 10339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:35791
I20260620 20:17:26.312794 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:17:26.322145 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.314769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.24.20.190:44237)
0620 20:17:26.314924 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:26.314927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:26.315151 (+ 224us) server_negotiation.cc:407] Connection header received
0620 20:17:26.315255 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:26.315259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:26.315320 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:26.315392 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:26.316124 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.316741 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.317588 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.317786 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.317863 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.319354 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:26.319364 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:26.319366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:26.319387 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:26.320584 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:26.321075 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:26.321078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:26.321079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:26.321158 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:26.321457 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:26.321459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:26.321460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:26.321638 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:26.321751 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:26.321961 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 20:17:26.322035 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 20:17:26.325012 10339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:17:26.327982 10339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:17:26.342217 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.336033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:41323 (local address 127.0.0.1:40766)
0620 20:17:26.336716 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:26.336764 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:26.336885 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:26.337813 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:26.337816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:26.337832 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:26.337918 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.337924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.339319 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.339322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.340788 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.340797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.341689 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.341693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.341794 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.341830 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 20:17:26.341945 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 20:17:26.342049 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":212,"thread_start_us":99,"threads_started":1}
I20260620 20:17:26.342454 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.336104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.24.20.130:34055)
0620 20:17:26.336523 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:26.336535 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:26.336852 (+ 317us) server_negotiation.cc:407] Connection header received
0620 20:17:26.337061 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:26.337066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:26.337280 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:26.337412 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:26.337984 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.339219 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.340993 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.341852 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.341963 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.342093 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:26.342180 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 20:17:26.342275 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":136,"threads_started":1}
I20260620 20:17:26.342543 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.336733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.24.20.131:41323)
0620 20:17:26.337395 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:26.337405 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:26.337424 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:17:26.337505 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:26.337510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:26.337690 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:26.337838 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:26.337999 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.339187 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.340951 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.341561 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.341691 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.341855 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:26.342290 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 20:17:26.342417 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":135,"threads_started":1}
I20260620 20:17:26.342597 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.336317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:41649 (local address 127.0.0.1:46862)
0620 20:17:26.337119 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:26.337133 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:26.337222 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:26.338121 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:26.338123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:26.338136 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:26.338203 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.338207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.339190 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.339193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.340900 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.340908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.342181 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.342185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.342269 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.342291 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:17:26.342378 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 20:17:26.342440 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":192,"thread_start_us":79,"threads_started":1}
I20260620 20:17:26.342881 10809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.335969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:34055 (local address 127.0.0.1:54310)
0620 20:17:26.336716 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:26.336764 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:26.336908 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:26.337680 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:26.337685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:26.337707 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:26.337848 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.337855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.339368 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.339371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.340841 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.340852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.341966 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.341970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.342047 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.342077 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 20:17:26.342148 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 20:17:26.342199 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":95,"thread_start_us":61,"threads_started":1}
I20260620 20:17:26.343037 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.336732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.24.20.129:41649)
0620 20:17:26.337172 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:26.337187 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:26.337208 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:17:26.337386 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:26.337393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:26.337597 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:26.337752 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:26.338315 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.339041 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.341034 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.341852 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.341963 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.342106 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:26.342799 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 20:17:26.342903 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":65,"threads_started":1}
I20260620 20:17:26.345630 10464 tablet_service.cc:1467] Processing CreateTablet for tablet 61d82ea92ac240da94bd42ea6ac16719 (DEFAULT_TABLE table=test-table [id=633ab99e8e7f42d4848de35fd0f3dd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:26.345685 10732 tablet_service.cc:1467] Processing CreateTablet for tablet 61d82ea92ac240da94bd42ea6ac16719 (DEFAULT_TABLE table=test-table [id=633ab99e8e7f42d4848de35fd0f3dd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:26.345890 10598 tablet_service.cc:1467] Processing CreateTablet for tablet 61d82ea92ac240da94bd42ea6ac16719 (DEFAULT_TABLE table=test-table [id=633ab99e8e7f42d4848de35fd0f3dd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:26.346634 10464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61d82ea92ac240da94bd42ea6ac16719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:26.346674 10598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61d82ea92ac240da94bd42ea6ac16719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:26.346674 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61d82ea92ac240da94bd42ea6ac16719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:26.352667 10815 tablet_bootstrap.cc:492] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6: Bootstrap starting.
I20260620 20:17:26.353801 10816 tablet_bootstrap.cc:492] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad: Bootstrap starting.
I20260620 20:17:26.354074 10817 tablet_bootstrap.cc:492] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63: Bootstrap starting.
I20260620 20:17:26.354971 10815 tablet_bootstrap.cc:654] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:26.355700 10817 tablet_bootstrap.cc:654] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:26.355832 10816 tablet_bootstrap.cc:654] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:26.356272 10815 log.cc:824] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:26.356832 10817 log.cc:824] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:26.357096 10816 log.cc:824] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:26.358587 10815 tablet_bootstrap.cc:492] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6: No bootstrap required, opened a new log
I20260620 20:17:26.358695 10817 tablet_bootstrap.cc:492] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63: No bootstrap required, opened a new log
I20260620 20:17:26.358737 10816 tablet_bootstrap.cc:492] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad: No bootstrap required, opened a new log
I20260620 20:17:26.358824 10815 ts_tablet_manager.cc:1397] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 20:17:26.358870 10816 ts_tablet_manager.cc:1397] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:17:26.358871 10817 ts_tablet_manager.cc:1397] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 20:17:26.361897 10815 raft_consensus.cc:348] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd4656f3f49423ebc2a86d610a12dad" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34055 } } peers { permanent_uuid: "2e092b59fbee4c37933513e82fc48c63" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41323 } } peers { permanent_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41649 } }
I20260620 20:17:26.362169 10815 raft_consensus.cc:374] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:26.362223 10815 raft_consensus.cc:729] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09764f01deb041ebbf05bbf53fe5a3a6, State: Initialized, Role: FOLLOWER
I20260620 20:17:26.362679 10815 consensus_queue.cc:260] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd4656f3f49423ebc2a86d610a12dad" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34055 } } peers { permanent_uuid: "2e092b59fbee4c37933513e82fc48c63" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41323 } } peers { permanent_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41649 } }
I20260620 20:17:26.362625 10817 raft_consensus.cc:348] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd4656f3f49423ebc2a86d610a12dad" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34055 } } peers { permanent_uuid: "2e092b59fbee4c37933513e82fc48c63" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41323 } } peers { permanent_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41649 } }
I20260620 20:17:26.362625 10816 raft_consensus.cc:348] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd4656f3f49423ebc2a86d610a12dad" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34055 } } peers { permanent_uuid: "2e092b59fbee4c37933513e82fc48c63" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41323 } } peers { permanent_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41649 } }
I20260620 20:17:26.362860 10817 raft_consensus.cc:374] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:26.362860 10816 raft_consensus.cc:374] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:26.362916 10817 raft_consensus.cc:729] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e092b59fbee4c37933513e82fc48c63, State: Initialized, Role: FOLLOWER
I20260620 20:17:26.362915 10816 raft_consensus.cc:729] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd4656f3f49423ebc2a86d610a12dad, State: Initialized, Role: FOLLOWER
I20260620 20:17:26.363551 10815 ts_tablet_manager.cc:1428] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 20:17:26.363466 10817 consensus_queue.cc:260] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd4656f3f49423ebc2a86d610a12dad" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34055 } } peers { permanent_uuid: "2e092b59fbee4c37933513e82fc48c63" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41323 } } peers { permanent_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41649 } }
I20260620 20:17:26.363484 10816 consensus_queue.cc:260] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd4656f3f49423ebc2a86d610a12dad" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34055 } } peers { permanent_uuid: "2e092b59fbee4c37933513e82fc48c63" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41323 } } peers { permanent_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41649 } }
I20260620 20:17:26.363924 10530 heartbeater.cc:503] Master 127.24.20.190:44237 was elected leader, sending a full tablet report...
I20260620 20:17:26.364482 10817 ts_tablet_manager.cc:1428] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 20:17:26.364482 10816 ts_tablet_manager.cc:1428] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 20:17:26.364509 10664 heartbeater.cc:503] Master 127.24.20.190:44237 was elected leader, sending a full tablet report...
I20260620 20:17:26.365410 10798 heartbeater.cc:503] Master 127.24.20.190:44237 was elected leader, sending a full tablet report...
W20260620 20:17:26.544314 10799 tablet.cc:2369] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:17:26.564663 10823 raft_consensus.cc:482] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:26.564908 10823 raft_consensus.cc:504] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd4656f3f49423ebc2a86d610a12dad" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34055 } } peers { permanent_uuid: "2e092b59fbee4c37933513e82fc48c63" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41323 } } peers { permanent_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41649 } }
I20260620 20:17:26.566072 10823 leader_election.cc:288] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dd4656f3f49423ebc2a86d610a12dad (127.24.20.130:34055), 09764f01deb041ebbf05bbf53fe5a3a6 (127.24.20.129:41649)
I20260620 20:17:26.568512 10821 raft_consensus.cc:482] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:26.568704 10821 raft_consensus.cc:504] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd4656f3f49423ebc2a86d610a12dad" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34055 } } peers { permanent_uuid: "2e092b59fbee4c37933513e82fc48c63" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41323 } } peers { permanent_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41649 } }
I20260620 20:17:26.569728 10821 leader_election.cc:288] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dd4656f3f49423ebc2a86d610a12dad (127.24.20.130:34055), 2e092b59fbee4c37933513e82fc48c63 (127.24.20.131:41323)
I20260620 20:17:26.570504 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.566316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:34055 (local address 127.24.20.131:42757)
0620 20:17:26.566430 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:26.566446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:26.566540 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:26.566843 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:26.566846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:26.566865 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:26.566996 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.567002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.567841 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.567844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.569074 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.569086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.570031 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.570035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.570155 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.570178 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:17:26.570252 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:17:26.570329 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 20:17:26.570986 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.566465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:44795 (local address 127.24.20.129:41649)
0620 20:17:26.566611 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:26.566617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:26.567027 (+ 410us) server_negotiation.cc:407] Connection header received
0620 20:17:26.567154 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:26.567162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:26.567204 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:26.567287 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:26.567560 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.568303 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.569666 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.570244 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.570360 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.570409 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:26.570813 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 20:17:26.570881 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:17:26.570981 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.566401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:42757 (local address 127.24.20.130:34055)
0620 20:17:26.566596 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:26.566602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:26.566621 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:17:26.566686 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:26.566688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:26.566729 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:26.566822 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:26.567112 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.567723 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.569207 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.569766 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.569885 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.569969 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:26.570780 (+ 811us) server_negotiation.cc:299] Negotiation successful
0620 20:17:26.570857 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 20:17:26.571024 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.566323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:41649 (local address 127.24.20.131:44795)
0620 20:17:26.566930 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:26.566944 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:26.567063 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:26.567301 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:26.567303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:26.567318 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:26.567448 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.567453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.568429 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.568432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.569551 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.569560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.570564 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.570569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.570679 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.570703 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:17:26.570786 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 20:17:26.570870 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":291,"thread_start_us":123,"threads_started":1}
I20260620 20:17:26.571559 10484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d82ea92ac240da94bd42ea6ac16719" candidate_uuid: "2e092b59fbee4c37933513e82fc48c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" is_pre_election: true
I20260620 20:17:26.571780 10484 raft_consensus.cc:2454] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e092b59fbee4c37933513e82fc48c63 in term 0.
I20260620 20:17:26.571753 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d82ea92ac240da94bd42ea6ac16719" candidate_uuid: "2e092b59fbee4c37933513e82fc48c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd4656f3f49423ebc2a86d610a12dad" is_pre_election: true
I20260620 20:17:26.572007 10618 raft_consensus.cc:2454] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e092b59fbee4c37933513e82fc48c63 in term 0.
I20260620 20:17:26.572248 10686 leader_election.cc:302] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09764f01deb041ebbf05bbf53fe5a3a6, 2e092b59fbee4c37933513e82fc48c63; no voters:
I20260620 20:17:26.572504 10823 raft_consensus.cc:2781] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:17:26.572572 10823 raft_consensus.cc:482] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:17:26.572649 10823 raft_consensus.cc:3037] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:26.573690 10823 raft_consensus.cc:504] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd4656f3f49423ebc2a86d610a12dad" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34055 } } peers { permanent_uuid: "2e092b59fbee4c37933513e82fc48c63" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41323 } } peers { permanent_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41649 } }
I20260620 20:17:26.574051 10823 leader_election.cc:288] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [CANDIDATE]: Term 1 election: Requested vote from peers 0dd4656f3f49423ebc2a86d610a12dad (127.24.20.130:34055), 09764f01deb041ebbf05bbf53fe5a3a6 (127.24.20.129:41649)
I20260620 20:17:26.574368 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d82ea92ac240da94bd42ea6ac16719" candidate_uuid: "2e092b59fbee4c37933513e82fc48c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd4656f3f49423ebc2a86d610a12dad"
I20260620 20:17:26.574415 10484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d82ea92ac240da94bd42ea6ac16719" candidate_uuid: "2e092b59fbee4c37933513e82fc48c63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09764f01deb041ebbf05bbf53fe5a3a6"
I20260620 20:17:26.574481 10618 raft_consensus.cc:3037] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:26.574523 10484 raft_consensus.cc:3037] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:26.575400 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.569867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:34055 (local address 127.24.20.129:55253)
0620 20:17:26.570361 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:26.570374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:26.570503 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:26.571519 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:26.571522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:26.571538 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:26.571670 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.571717 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.572676 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.572679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.573943 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.573953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.574867 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.574870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.575026 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.575099 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 20:17:26.575171 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 20:17:26.575254 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":73,"thread_start_us":111,"threads_started":1}
I20260620 20:17:26.575405 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.570406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:59479 (local address 127.24.20.131:41323)
0620 20:17:26.570509 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:26.570513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:26.570875 (+ 362us) server_negotiation.cc:407] Connection header received
0620 20:17:26.571033 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:26.571039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:26.571084 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:26.571136 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:26.572010 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.572709 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.574127 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.574686 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.574765 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.574806 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:26.575250 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 20:17:26.575315 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:17:26.575726 10484 raft_consensus.cc:2454] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e092b59fbee4c37933513e82fc48c63 in term 1.
I20260620 20:17:26.575731 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.570220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:55253 (local address 127.24.20.130:34055)
0620 20:17:26.571131 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:26.571136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:26.571147 (+ 11us) server_negotiation.cc:407] Connection header received
0620 20:17:26.571188 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:26.571190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:26.571232 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:26.571299 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:26.571809 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.572534 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.574098 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.574663 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.574769 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.574806 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:26.575541 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 20:17:26.575624 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":65,"threads_started":1}
I20260620 20:17:26.575727 10618 raft_consensus.cc:2454] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e092b59fbee4c37933513e82fc48c63 in term 1.
I20260620 20:17:26.576067 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.569998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:41323 (local address 127.24.20.129:59479)
0620 20:17:26.570807 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:26.570819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:26.570931 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:26.571428 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:26.571431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:26.571450 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:26.571815 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.571820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.572891 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.572895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.574004 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:26.574014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.574884 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.574887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:26.575064 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.575143 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 20:17:26.575719 (+ 576us) client_negotiation.cc:240] Negotiation successful
0620 20:17:26.575890 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":100,"thread_start_us":99,"threads_started":1}
I20260620 20:17:26.576095 10686 leader_election.cc:302] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09764f01deb041ebbf05bbf53fe5a3a6, 2e092b59fbee4c37933513e82fc48c63; no voters:
I20260620 20:17:26.576259 10823 raft_consensus.cc:2781] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:26.576399 10618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d82ea92ac240da94bd42ea6ac16719" candidate_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd4656f3f49423ebc2a86d610a12dad" is_pre_election: true
I20260620 20:17:26.576488 10823 raft_consensus.cc:686] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [term 1 LEADER]: Becoming Leader. State: Replica: 2e092b59fbee4c37933513e82fc48c63, State: Running, Role: LEADER
I20260620 20:17:26.576635 10618 raft_consensus.cc:2379] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09764f01deb041ebbf05bbf53fe5a3a6 in current term 1: Already voted for candidate 2e092b59fbee4c37933513e82fc48c63 in this term.
I20260620 20:17:26.576705 10752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d82ea92ac240da94bd42ea6ac16719" candidate_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e092b59fbee4c37933513e82fc48c63" is_pre_election: true
I20260620 20:17:26.576840 10823 consensus_queue.cc:237] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd4656f3f49423ebc2a86d610a12dad" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34055 } } peers { permanent_uuid: "2e092b59fbee4c37933513e82fc48c63" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41323 } } peers { permanent_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41649 } }
I20260620 20:17:26.578437 10417 leader_election.cc:302] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09764f01deb041ebbf05bbf53fe5a3a6; no voters: 0dd4656f3f49423ebc2a86d610a12dad, 2e092b59fbee4c37933513e82fc48c63
I20260620 20:17:26.578667 10821 raft_consensus.cc:2726] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 20:17:26.579996 10338 catalog_manager.cc:5495] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e092b59fbee4c37933513e82fc48c63 (127.24.20.131). New cstate: current_term: 1 leader_uuid: "2e092b59fbee4c37933513e82fc48c63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd4656f3f49423ebc2a86d610a12dad" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e092b59fbee4c37933513e82fc48c63" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 41323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41649 } health_report { overall_health: UNKNOWN } } }
W20260620 20:17:26.581581 10665 tablet.cc:2369] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:17:26.612700 10531 tablet.cc:2369] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:17:26.633569 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.629969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.24.20.131:41323)
0620 20:17:26.630123 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:26.630127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:26.630142 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:17:26.630254 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:26.630256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:26.630311 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:26.630429 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:17:26.630697 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.631203 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.632060 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.632283 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.632393 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.632721 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:17:26.633118 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:17:26.633278 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:26.633376 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 20:17:26.633411 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:17:26.639142 10618 raft_consensus.cc:1261] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad [term 1 FOLLOWER]: Refusing update from remote peer 2e092b59fbee4c37933513e82fc48c63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:26.639199 10484 raft_consensus.cc:1261] T 61d82ea92ac240da94bd42ea6ac16719 P 09764f01deb041ebbf05bbf53fe5a3a6 [term 1 FOLLOWER]: Refusing update from remote peer 2e092b59fbee4c37933513e82fc48c63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:26.639804 10823 consensus_queue.cc:1035] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dd4656f3f49423ebc2a86d610a12dad" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 34055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:26.640022 10828 consensus_queue.cc:1035] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09764f01deb041ebbf05bbf53fe5a3a6" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 41649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:26.651199 10833 mvcc.cc:205] Tried to move back new op lower bound from 7299017304626892800 to 7299017304381370368. Current Snapshot: MvccSnapshot[applied={T|T < 7299017304626892800 or (T in {7299017304626892800})}]
I20260620 20:17:26.655562 10339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:26.655702 10339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:26.657922 10339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cca064fc502459f850f6e7beef07a31: Sending DeleteTablet for 3 replicas of tablet 61d82ea92ac240da94bd42ea6ac16719
I20260620 20:17:26.658740 10598 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d82ea92ac240da94bd42ea6ac16719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:26 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:54310
I20260620 20:17:26.658731 10732 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d82ea92ac240da94bd42ea6ac16719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:26 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:40766
I20260620 20:17:26.659221 10842 tablet_replica.cc:330] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63: stopping tablet replica
I20260620 20:17:26.659384 10464 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d82ea92ac240da94bd42ea6ac16719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:26 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:46862
I20260620 20:17:26.659675 10842 raft_consensus.cc:2229] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:17:26.659919 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 10401
I20260620 20:17:26.660164 10842 raft_consensus.cc:2258] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:26.660898 10841 tablet_replica.cc:330] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad: stopping tablet replica
I20260620 20:17:26.661209 10841 raft_consensus.cc:2229] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:17:26.661505 10841 raft_consensus.cc:2258] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:26.663009 10841 ts_tablet_manager.cc:1905] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:17:26.663394 10842 ts_tablet_manager.cc:1905] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:17:26.666044 10842 ts_tablet_manager.cc:1918] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:26.666044 10841 ts_tablet_manager.cc:1918] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:26.666152 10842 log.cc:1196] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/61d82ea92ac240da94bd42ea6ac16719
I20260620 20:17:26.666285 10841 log.cc:1196] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/61d82ea92ac240da94bd42ea6ac16719
I20260620 20:17:26.666554 10842 ts_tablet_manager.cc:1939] T 61d82ea92ac240da94bd42ea6ac16719 P 2e092b59fbee4c37933513e82fc48c63: Deleting consensus metadata
I20260620 20:17:26.666745 10841 ts_tablet_manager.cc:1939] T 61d82ea92ac240da94bd42ea6ac16719 P 0dd4656f3f49423ebc2a86d610a12dad: Deleting consensus metadata
I20260620 20:17:26.667811 10323 catalog_manager.cc:4841] TS 0dd4656f3f49423ebc2a86d610a12dad (127.24.20.130:34055): tablet 61d82ea92ac240da94bd42ea6ac16719 (table test-table [id=633ab99e8e7f42d4848de35fd0f3dd01]) successfully deleted
I20260620 20:17:26.668026 10326 catalog_manager.cc:4841] TS 2e092b59fbee4c37933513e82fc48c63 (127.24.20.131:41323): tablet 61d82ea92ac240da94bd42ea6ac16719 (table test-table [id=633ab99e8e7f42d4848de35fd0f3dd01]) successfully deleted
W20260620 20:17:26.670912 10324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:41649 (error 108)
I20260620 20:17:26.671106 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 10535
I20260620 20:17:26.672391 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.672087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:41649 (local address 127.0.0.1:55302)
0620 20:17:26.672248 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:26.672311 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260620 20:17:26.672525 10324 catalog_manager.cc:4584] TS 09764f01deb041ebbf05bbf53fe5a3a6 (127.24.20.129:41649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61d82ea92ac240da94bd42ea6ac16719: Network error: Client connection negotiation failed: client connection to 127.24.20.129:41649: connect: Connection refused (error 111)
I20260620 20:17:26.677034 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 10669
I20260620 20:17:26.682435 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 10307
2026-06-20T20:17:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2432 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T20:17:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:26Z Disabled control of system clock
I20260620 20:17:26.710486 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:41407
--webserver_interface=127.24.20.190
--webserver_port=0
--builtin_ntp_servers=127.24.20.148:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:41407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:17:26.815860 10848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:26.816099 10848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:26.816133 10848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:26.819962 10848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:26.820015 10848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:26.820030 10848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:26.820047 10848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:26.820068 10848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:26.824628 10848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:41407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:41407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:26.825554 10848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:26.826815 10848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:26.835050 10854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:26.835079 10856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:26.835070 10853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:26.835554 10848 server_base.cc:1034] running on GCE node
I20260620 20:17:26.836021 10848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:26.837154 10848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:26.838325 10848 hybrid_clock.cc:648] HybridClock initialized: now 1781986646838306 us; error 29 us; skew 500 ppm
I20260620 20:17:26.839988 10848 webserver.cc:460] Webserver started at http://127.24.20.190:41771/ using document root <none> and password file <none>
I20260620 20:17:26.840512 10848 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:26.840569 10848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:26.840823 10848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:26.842793 10848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "4f46c6635aa04ed5968e3d24d2dea4c3"
format_stamp: "Formatted at 2026-06-20 20:17:26 on dist-test-slave-pflr"
server_key: "334c81cb4d5c2a8a33f7f8091b8ea511"
server_key_iv: "9d15da3a1f1f1defc49bb94cd7784459"
server_key_version: "encryptionkey@0"
I20260620 20:17:26.843289 10848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "4f46c6635aa04ed5968e3d24d2dea4c3"
format_stamp: "Formatted at 2026-06-20 20:17:26 on dist-test-slave-pflr"
server_key: "334c81cb4d5c2a8a33f7f8091b8ea511"
server_key_iv: "9d15da3a1f1f1defc49bb94cd7784459"
server_key_version: "encryptionkey@0"
I20260620 20:17:26.846767 10848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:17:26.848979 10862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:26.850133 10848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:17:26.850225 10848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "4f46c6635aa04ed5968e3d24d2dea4c3"
format_stamp: "Formatted at 2026-06-20 20:17:26 on dist-test-slave-pflr"
server_key: "334c81cb4d5c2a8a33f7f8091b8ea511"
server_key_iv: "9d15da3a1f1f1defc49bb94cd7784459"
server_key_version: "encryptionkey@0"
I20260620 20:17:26.850301 10848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:26.866122 10848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:26.866727 10848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:26.866878 10848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:26.873553 10848 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:41407
I20260620 20:17:26.873616 10914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:41407 every 8 connection(s)
I20260620 20:17:26.874460 10848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:26.875715 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 10848
I20260620 20:17:26.875842 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:26.876204 24658 external_mini_cluster.cc:1428] Setting key 1966abe1677600a019ddd22331a48f3b
I20260620 20:17:26.878088 10915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:26.884306 10915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3: Bootstrap starting.
I20260620 20:17:26.885747 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:26.877649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51078 (local address 127.24.20.190:41407)
0620 20:17:26.878333 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:26.878348 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:26.878381 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:17:26.879000 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:26.879014 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:26.879381 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:26.879720 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:26.880177 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.881356 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.882585 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:26.883048 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:26.883220 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:26.884076 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:26.884081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:26.884092 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:26.884490 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:26.884542 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:26.884562 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:26.884702 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:26.884820 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 20:17:26.885099 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":133,"threads_started":1}
I20260620 20:17:26.886461 10915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:26.887164 10915 log.cc:824] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:26.888964 10915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3: No bootstrap required, opened a new log
I20260620 20:17:26.892271 10915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f46c6635aa04ed5968e3d24d2dea4c3" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41407 } }
I20260620 20:17:26.892496 10915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:26.892583 10915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f46c6635aa04ed5968e3d24d2dea4c3, State: Initialized, Role: FOLLOWER
I20260620 20:17:26.893025 10915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f46c6635aa04ed5968e3d24d2dea4c3" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41407 } }
I20260620 20:17:26.893170 10915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:26.893241 10915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:26.893347 10915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:26.894109 10915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f46c6635aa04ed5968e3d24d2dea4c3" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41407 } }
I20260620 20:17:26.894378 10915 leader_election.cc:302] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f46c6635aa04ed5968e3d24d2dea4c3; no voters:
I20260620 20:17:26.894608 10915 leader_election.cc:288] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:26.894707 10920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:26.894951 10920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 4f46c6635aa04ed5968e3d24d2dea4c3, State: Running, Role: LEADER
I20260620 20:17:26.895226 10920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f46c6635aa04ed5968e3d24d2dea4c3" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41407 } }
I20260620 20:17:26.895699 10915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:26.896767 10922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f46c6635aa04ed5968e3d24d2dea4c3. Latest consensus state: current_term: 1 leader_uuid: "4f46c6635aa04ed5968e3d24d2dea4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f46c6635aa04ed5968e3d24d2dea4c3" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41407 } } }
I20260620 20:17:26.896899 10922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:26.897181 10921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f46c6635aa04ed5968e3d24d2dea4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f46c6635aa04ed5968e3d24d2dea4c3" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 41407 } } }
I20260620 20:17:26.897267 10921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:26.897358 10929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:26.900075 10929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:26.905728 10929 catalog_manager.cc:1347] Generated new cluster ID: 347a024d830140019944b98278f399cf
I20260620 20:17:26.905789 10929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:26.915175 10929 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:26.916299 10929 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:26.921552 10929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3: Generated new TSK 0
I20260620 20:17:26.922259 10929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:26.928115 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41407
--builtin_ntp_servers=127.24.20.148:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:17:27.037469 10939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:27.037715 10939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:27.037752 10939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:27.041751 10939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:27.041801 10939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:27.041855 10939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:17:27.046496 10939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:27.047513 10939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:27.048928 10939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:27.058017 10947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:27.058023 10944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:27.058187 10939 server_base.cc:1034] running on GCE node
W20260620 20:17:27.058017 10945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:27.058671 10939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:27.059262 10939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:27.060416 10939 hybrid_clock.cc:648] HybridClock initialized: now 1781986647060402 us; error 26 us; skew 500 ppm
I20260620 20:17:27.062238 10939 webserver.cc:460] Webserver started at http://127.24.20.129:32925/ using document root <none> and password file <none>
I20260620 20:17:27.062793 10939 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:27.062852 10939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:27.063017 10939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:27.065055 10939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "26dbc9e46455412ba550ef7b0b1ce66f"
format_stamp: "Formatted at 2026-06-20 20:17:27 on dist-test-slave-pflr"
server_key: "1571f63b6e3280c2e6b1ad59393240bf"
server_key_iv: "0a754936202f956216bbe55e43bffaef"
server_key_version: "encryptionkey@0"
I20260620 20:17:27.065567 10939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "26dbc9e46455412ba550ef7b0b1ce66f"
format_stamp: "Formatted at 2026-06-20 20:17:27 on dist-test-slave-pflr"
server_key: "1571f63b6e3280c2e6b1ad59393240bf"
server_key_iv: "0a754936202f956216bbe55e43bffaef"
server_key_version: "encryptionkey@0"
I20260620 20:17:27.069064 10939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:27.071275 10953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:27.072435 10939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:27.072564 10939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "26dbc9e46455412ba550ef7b0b1ce66f"
format_stamp: "Formatted at 2026-06-20 20:17:27 on dist-test-slave-pflr"
server_key: "1571f63b6e3280c2e6b1ad59393240bf"
server_key_iv: "0a754936202f956216bbe55e43bffaef"
server_key_version: "encryptionkey@0"
I20260620 20:17:27.072723 10939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:27.090891 10939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:27.091554 10939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:27.091737 10939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:27.092303 10939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:27.093274 10939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:27.093343 10939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:27.093417 10939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:27.093458 10939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:27.102248 10939 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:37647
I20260620 20:17:27.102274 11066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:37647 every 8 connection(s)
I20260620 20:17:27.103206 10939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:17:27.104363 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 10939
I20260620 20:17:27.104487 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:17:27.104815 24658 external_mini_cluster.cc:1428] Setting key 3f5bdc114418aae8cc9b877313186a95
I20260620 20:17:27.106988 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41407
--builtin_ntp_servers=127.24.20.148:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:17:27.114063 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.105172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:40707 (local address 127.24.20.190:41407)
0620 20:17:27.105323 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:27.105326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:27.106630 (+ 1304us) server_negotiation.cc:407] Connection header received
0620 20:17:27.108682 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:27.108687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:27.108752 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:27.108852 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:27.109701 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.110602 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.111790 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.112237 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.112366 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.113444 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:27.113450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:27.113457 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:27.113476 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:27.113503 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:27.113514 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:27.113601 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:27.113747 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 20:17:27.113848 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:17:27.114723 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.105700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:41407 (local address 127.24.20.129:40707)
0620 20:17:27.106479 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:27.106531 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:27.108386 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:27.109005 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:27.109011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:27.109092 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:27.109552 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.109563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.110726 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.110729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.111664 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.111674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.112384 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.112387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.112517 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.113197 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:27.113199 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:27.113236 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:27.113297 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:27.113607 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:27.113614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:27.113618 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:17:27.113707 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 20:17:27.113911 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":139,"threads_started":1}
I20260620 20:17:27.115906 11067 heartbeater.cc:347] Connected to a master server at 127.24.20.190:41407
I20260620 20:17:27.116115 11067 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:27.116644 11067 heartbeater.cc:511] Master 127.24.20.190:41407 requested a full tablet report, sending...
I20260620 20:17:27.118049 10879 ts_manager.cc:194] Registered new tserver with Master: 26dbc9e46455412ba550ef7b0b1ce66f (127.24.20.129:37647)
I20260620 20:17:27.119163 10879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.20.129:40707
W20260620 20:17:27.218014 11070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:27.218272 11070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:27.218313 11070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:27.222771 11070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:27.222826 11070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:27.222885 11070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:17:27.227659 11070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:27.228641 11070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:27.229930 11070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:17:27.239084 11070 server_base.cc:1034] running on GCE node
W20260620 20:17:27.239014 11076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:27.239046 11075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:27.239061 11078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:27.239696 11070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:27.240289 11070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:27.241439 11070 hybrid_clock.cc:648] HybridClock initialized: now 1781986647241410 us; error 43 us; skew 500 ppm
I20260620 20:17:27.243295 11070 webserver.cc:460] Webserver started at http://127.24.20.130:42559/ using document root <none> and password file <none>
I20260620 20:17:27.243834 11070 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:27.243893 11070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:27.244143 11070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:27.246109 11070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17"
format_stamp: "Formatted at 2026-06-20 20:17:27 on dist-test-slave-pflr"
server_key: "b3ad8885cb7b7ebd51fd5110d54d5f7c"
server_key_iv: "2ed778e7cbf53e9502230da794d29e16"
server_key_version: "encryptionkey@0"
I20260620 20:17:27.246642 11070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17"
format_stamp: "Formatted at 2026-06-20 20:17:27 on dist-test-slave-pflr"
server_key: "b3ad8885cb7b7ebd51fd5110d54d5f7c"
server_key_iv: "2ed778e7cbf53e9502230da794d29e16"
server_key_version: "encryptionkey@0"
I20260620 20:17:27.250088 11070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:17:27.252280 11084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:27.253557 11070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:27.253676 11070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17"
format_stamp: "Formatted at 2026-06-20 20:17:27 on dist-test-slave-pflr"
server_key: "b3ad8885cb7b7ebd51fd5110d54d5f7c"
server_key_iv: "2ed778e7cbf53e9502230da794d29e16"
server_key_version: "encryptionkey@0"
I20260620 20:17:27.253779 11070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:27.275669 11070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:27.276284 11070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:27.276471 11070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:27.277062 11070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:27.277912 11070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:27.277977 11070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:27.278043 11070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:27.278093 11070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:27.286589 11070 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:44165
I20260620 20:17:27.286617 11197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:44165 every 8 connection(s)
I20260620 20:17:27.287403 11070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:17:27.294054 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 11070
I20260620 20:17:27.294167 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:17:27.294481 24658 external_mini_cluster.cc:1428] Setting key 9987a2afe15154977bd77b3aff677556
I20260620 20:17:27.296195 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.289046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:52459 (local address 127.24.20.190:41407)
0620 20:17:27.289212 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:27.289216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:27.290334 (+ 1118us) server_negotiation.cc:407] Connection header received
0620 20:17:27.291607 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:27.291610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:27.291656 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:27.291729 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:27.292486 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.293014 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.294067 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.294417 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.294528 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.295651 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:27.295654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:27.295656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:27.295667 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:27.295685 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:27.295693 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:27.295785 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:27.296000 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 20:17:27.296058 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:17:27.296626 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41407
--builtin_ntp_servers=127.24.20.148:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:17:27.297083 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.289439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:41407 (local address 127.24.20.130:52459)
0620 20:17:27.290178 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:27.290216 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:27.291428 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:27.291853 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:27.291859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:27.291936 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:27.292373 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.292384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.293117 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.293120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.293952 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.293960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.294585 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.294593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.294736 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.295396 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:27.295398 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:27.295447 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:27.295515 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:27.295800 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:27.295810 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:27.295820 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 20:17:27.295951 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 20:17:27.296172 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":227,"threads_started":1}
I20260620 20:17:27.298132 11198 heartbeater.cc:347] Connected to a master server at 127.24.20.190:41407
I20260620 20:17:27.298333 11198 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:27.298796 11198 heartbeater.cc:511] Master 127.24.20.190:41407 requested a full tablet report, sending...
I20260620 20:17:27.299726 10879 ts_manager.cc:194] Registered new tserver with Master: 8ce5fdfca8704a4b8d7cf973a80d0a17 (127.24.20.130:44165)
I20260620 20:17:27.300267 10879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.20.130:52459
W20260620 20:17:27.404969 11201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:27.405237 11201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:27.405300 11201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:27.409266 11201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:27.409341 11201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:27.409423 11201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:17:27.414065 11201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:41407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:27.415061 11201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:27.416359 11201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:27.425602 11209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:27.425611 11206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:27.425704 11201 server_base.cc:1034] running on GCE node
W20260620 20:17:27.425611 11207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:27.426222 11201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:27.426779 11201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:27.427956 11201 hybrid_clock.cc:648] HybridClock initialized: now 1781986647427951 us; error 30 us; skew 500 ppm
I20260620 20:17:27.429718 11201 webserver.cc:460] Webserver started at http://127.24.20.131:40877/ using document root <none> and password file <none>
I20260620 20:17:27.430250 11201 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:27.430339 11201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:27.430562 11201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:27.432508 11201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "10b4a9de87c34374a22f92d632218d41"
format_stamp: "Formatted at 2026-06-20 20:17:27 on dist-test-slave-pflr"
server_key: "4d0d80550a80b0e5960dba3b89138805"
server_key_iv: "f5711fa5a2d0b8b1a9fd6ecd2d0a6b5b"
server_key_version: "encryptionkey@0"
I20260620 20:17:27.433089 11201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "10b4a9de87c34374a22f92d632218d41"
format_stamp: "Formatted at 2026-06-20 20:17:27 on dist-test-slave-pflr"
server_key: "4d0d80550a80b0e5960dba3b89138805"
server_key_iv: "f5711fa5a2d0b8b1a9fd6ecd2d0a6b5b"
server_key_version: "encryptionkey@0"
I20260620 20:17:27.436699 11201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:27.438890 11215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:27.440033 11201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:27.440157 11201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "10b4a9de87c34374a22f92d632218d41"
format_stamp: "Formatted at 2026-06-20 20:17:27 on dist-test-slave-pflr"
server_key: "4d0d80550a80b0e5960dba3b89138805"
server_key_iv: "f5711fa5a2d0b8b1a9fd6ecd2d0a6b5b"
server_key_version: "encryptionkey@0"
I20260620 20:17:27.440259 11201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:27.453131 11201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:27.453704 11201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:27.453881 11201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:27.454403 11201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:27.455394 11201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:27.455463 11201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:27.455524 11201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:27.455567 11201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:27.464625 11201 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:33065
I20260620 20:17:27.464700 11329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:33065 every 8 connection(s)
I20260620 20:17:27.465754 11201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:17:27.472859 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 11201
I20260620 20:17:27.472978 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:17:27.473285 24658 external_mini_cluster.cc:1428] Setting key 6727aa7f20aa9acfbc279011a339a22f
I20260620 20:17:27.474761 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.467592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:38379 (local address 127.24.20.190:41407)
0620 20:17:27.467736 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:27.467741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:27.468845 (+ 1104us) server_negotiation.cc:407] Connection header received
0620 20:17:27.470132 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:27.470135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:27.470179 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:27.470252 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:27.471052 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.471562 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.472643 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.472866 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.472980 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.474279 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:27.474282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:27.474284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:27.474294 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:27.474311 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:27.474318 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:27.474396 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:27.474567 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 20:17:27.474626 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:17:27.475719 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.468032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:41407 (local address 127.24.20.131:38379)
0620 20:17:27.468701 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:27.468745 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:27.469947 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:27.470397 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:27.470403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:27.470481 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:27.470929 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.470940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.471659 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.471662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.472504 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.472512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.473030 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.473037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.473231 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.473939 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:27.473943 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:27.474004 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:27.474077 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:27.474410 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:27.474420 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:27.474426 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:17:27.474566 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 20:17:27.474824 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20260620 20:17:27.476775 11330 heartbeater.cc:347] Connected to a master server at 127.24.20.190:41407
I20260620 20:17:27.477017 11330 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:27.477451 11330 heartbeater.cc:511] Master 127.24.20.190:41407 requested a full tablet report, sending...
I20260620 20:17:27.478302 10879 ts_manager.cc:194] Registered new tserver with Master: 10b4a9de87c34374a22f92d632218d41 (127.24.20.131:33065)
I20260620 20:17:27.478801 10879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.20.131:38379
I20260620 20:17:27.487000 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:17:27.492512 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.488872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.24.20.190:41407)
0620 20:17:27.488986 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:27.488989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:27.489180 (+ 191us) server_negotiation.cc:407] Connection header received
0620 20:17:27.489260 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:27.489261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:27.489295 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:27.489359 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:27.489589 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.490161 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.491018 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.491223 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.491295 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.492158 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:27.492161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:27.492163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:27.492174 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:27.492190 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:27.492197 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:27.492293 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:27.492392 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 20:17:27.492420 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:17:27.495093 10879 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:17:27.498273 10879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:17:27.511518 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.506049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.24.20.131:33065)
0620 20:17:27.506494 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:27.506507 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:27.506539 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:17:27.506628 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:27.506634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:27.506854 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:27.507019 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:27.507355 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.508524 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.510135 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.510839 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.510963 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.511153 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:27.511240 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 20:17:27.511343 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":149,"threads_started":1}
I20260620 20:17:27.511554 11343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.506178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:37647 (local address 127.0.0.1:39348)
0620 20:17:27.506682 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:27.506695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:27.506764 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:27.507644 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:27.507647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:27.507666 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:27.507770 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.507775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.509126 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.509131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.510022 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.510031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.511056 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.511059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.511146 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.511170 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:17:27.511217 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 20:17:27.511385 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":32,"thread_start_us":85,"threads_started":1}
I20260620 20:17:27.511570 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.505914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:33065 (local address 127.0.0.1:55708)
0620 20:17:27.506278 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:27.506327 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:27.506457 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:27.507032 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:27.507039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:27.507065 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:27.507214 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.507221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.508681 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.508684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.509984 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.509994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.510959 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.510962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.511071 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.511109 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 20:17:27.511215 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:17:27.511385 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":44,"thread_start_us":109,"threads_started":1}
I20260620 20:17:27.512475 11344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.506667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:44165 (local address 127.0.0.1:60802)
0620 20:17:27.507623 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:27.507635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:27.507787 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:27.508310 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:27.508312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:27.508325 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:27.508383 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.508386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.509783 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.509786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.510880 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.510893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.512128 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.512131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.512216 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.512239 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:17:27.512300 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 20:17:27.512358 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":90,"threads_started":1}
I20260620 20:17:27.512739 11346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.507080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60802 (local address 127.24.20.130:44165)
0620 20:17:27.507485 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:27.507498 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:27.507717 (+ 219us) server_negotiation.cc:407] Connection header received
0620 20:17:27.507945 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:27.507949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:27.508090 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:27.508183 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:27.508492 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.509658 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.511031 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.512005 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.512135 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.512312 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:27.512363 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 20:17:27.512452 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":132,"threads_started":1}
I20260620 20:17:27.512744 11345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.506723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.24.20.129:37647)
0620 20:17:27.507175 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:27.507187 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:27.507212 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:17:27.507290 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:27.507296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:27.507490 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:27.507908 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:27.508037 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.509007 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.510163 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.510930 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.512151 (+ 1221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.512312 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:27.512363 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 20:17:27.512452 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":118,"threads_started":1}
I20260620 20:17:27.514621 11264 tablet_service.cc:1467] Processing CreateTablet for tablet 8a98e8cb108443a692879da29ff7a49b (DEFAULT_TABLE table=test-table [id=36b45e8b707d4280aa0303e8e81a090a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:27.514860 11001 tablet_service.cc:1467] Processing CreateTablet for tablet 8a98e8cb108443a692879da29ff7a49b (DEFAULT_TABLE table=test-table [id=36b45e8b707d4280aa0303e8e81a090a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:27.515579 11264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a98e8cb108443a692879da29ff7a49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:27.515759 11001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a98e8cb108443a692879da29ff7a49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:27.519167 11132 tablet_service.cc:1467] Processing CreateTablet for tablet 8a98e8cb108443a692879da29ff7a49b (DEFAULT_TABLE table=test-table [id=36b45e8b707d4280aa0303e8e81a090a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:27.520144 11132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a98e8cb108443a692879da29ff7a49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:27.521540 11347 tablet_bootstrap.cc:492] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41: Bootstrap starting.
I20260620 20:17:27.523115 11348 tablet_bootstrap.cc:492] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f: Bootstrap starting.
I20260620 20:17:27.523980 11347 tablet_bootstrap.cc:654] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:27.524752 11347 log.cc:824] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:27.525197 11348 tablet_bootstrap.cc:654] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:27.526015 11348 log.cc:824] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:27.526551 11347 tablet_bootstrap.cc:492] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41: No bootstrap required, opened a new log
I20260620 20:17:27.526757 11347 ts_tablet_manager.cc:1397] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:17:27.527091 11351 tablet_bootstrap.cc:492] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17: Bootstrap starting.
I20260620 20:17:27.528055 11348 tablet_bootstrap.cc:492] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f: No bootstrap required, opened a new log
I20260620 20:17:27.528294 11348 ts_tablet_manager.cc:1397] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:17:27.529070 11351 tablet_bootstrap.cc:654] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:27.529834 11351 log.cc:824] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:27.530980 11347 raft_consensus.cc:348] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b4a9de87c34374a22f92d632218d41" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33065 } } peers { permanent_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44165 } } peers { permanent_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37647 } }
I20260620 20:17:27.531236 11347 raft_consensus.cc:374] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:27.531293 11347 raft_consensus.cc:729] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b4a9de87c34374a22f92d632218d41, State: Initialized, Role: FOLLOWER
I20260620 20:17:27.531701 11351 tablet_bootstrap.cc:492] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17: No bootstrap required, opened a new log
I20260620 20:17:27.531730 11347 consensus_queue.cc:260] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b4a9de87c34374a22f92d632218d41" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33065 } } peers { permanent_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44165 } } peers { permanent_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37647 } }
I20260620 20:17:27.531873 11351 ts_tablet_manager.cc:1397] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:17:27.531898 11348 raft_consensus.cc:348] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b4a9de87c34374a22f92d632218d41" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33065 } } peers { permanent_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44165 } } peers { permanent_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37647 } }
I20260620 20:17:27.532152 11348 raft_consensus.cc:374] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:27.532212 11348 raft_consensus.cc:729] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26dbc9e46455412ba550ef7b0b1ce66f, State: Initialized, Role: FOLLOWER
I20260620 20:17:27.532483 11330 heartbeater.cc:503] Master 127.24.20.190:41407 was elected leader, sending a full tablet report...
I20260620 20:17:27.532699 11348 consensus_queue.cc:260] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b4a9de87c34374a22f92d632218d41" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33065 } } peers { permanent_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44165 } } peers { permanent_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37647 } }
I20260620 20:17:27.532756 11347 ts_tablet_manager.cc:1428] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 20:17:27.533322 11067 heartbeater.cc:503] Master 127.24.20.190:41407 was elected leader, sending a full tablet report...
I20260620 20:17:27.533524 11348 ts_tablet_manager.cc:1428] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260620 20:17:27.537142 11351 raft_consensus.cc:348] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b4a9de87c34374a22f92d632218d41" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33065 } } peers { permanent_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44165 } } peers { permanent_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37647 } }
I20260620 20:17:27.537410 11351 raft_consensus.cc:374] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:27.537483 11351 raft_consensus.cc:729] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce5fdfca8704a4b8d7cf973a80d0a17, State: Initialized, Role: FOLLOWER
I20260620 20:17:27.537885 11351 consensus_queue.cc:260] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b4a9de87c34374a22f92d632218d41" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33065 } } peers { permanent_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44165 } } peers { permanent_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37647 } }
I20260620 20:17:27.538439 11198 heartbeater.cc:503] Master 127.24.20.190:41407 was elected leader, sending a full tablet report...
I20260620 20:17:27.538743 11351 ts_tablet_manager.cc:1428] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260620 20:17:27.604318 11068 tablet.cc:2369] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:17:27.622503 11355 raft_consensus.cc:482] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:27.622759 11355 raft_consensus.cc:504] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b4a9de87c34374a22f92d632218d41" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33065 } } peers { permanent_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44165 } } peers { permanent_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37647 } }
I20260620 20:17:27.623960 11355 leader_election.cc:288] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b4a9de87c34374a22f92d632218d41 (127.24.20.131:33065), 26dbc9e46455412ba550ef7b0b1ce66f (127.24.20.129:37647)
I20260620 20:17:27.626755 11353 raft_consensus.cc:482] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:27.626950 11353 raft_consensus.cc:504] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b4a9de87c34374a22f92d632218d41" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33065 } } peers { permanent_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44165 } } peers { permanent_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37647 } }
I20260620 20:17:27.628556 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.624202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:37647 (local address 127.24.20.130:41597)
0620 20:17:27.624301 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:27.624316 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:27.624412 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:27.624884 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:27.624887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:27.624904 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:27.625053 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.625061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.626105 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.626108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.627450 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.627461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.628106 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.628109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.628207 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.628236 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:17:27.628324 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 20:17:27.628397 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":34}
I20260620 20:17:27.628798 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.624349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:49745 (local address 127.24.20.131:33065)
0620 20:17:27.624508 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:27.624513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:27.624811 (+ 298us) server_negotiation.cc:407] Connection header received
0620 20:17:27.624970 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:27.624973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:27.625026 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:27.625092 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:27.625455 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.626152 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.627582 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.628209 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.628330 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.628380 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:27.628572 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 20:17:27.628670 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:17:27.629017 11345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.624432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:41597 (local address 127.24.20.129:37647)
0620 20:17:27.624558 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:27.624562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:27.624584 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:17:27.624678 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:27.624681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:27.624737 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:27.624864 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:27.625188 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.625984 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.627623 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.627989 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.628112 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.628163 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:27.628826 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 20:17:27.628907 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:17:27.629117 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.624202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:33065 (local address 127.24.20.130:49745)
0620 20:17:27.624718 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:27.624728 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:27.624833 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:27.625228 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:27.625231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:27.625245 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:27.625339 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.625345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.626268 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.626271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.627466 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.627472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.628339 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.628342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.628424 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.628443 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 20:17:27.628906 (+ 463us) client_negotiation.cc:240] Negotiation successful
0620 20:17:27.628974 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":172,"thread_start_us":122,"threads_started":1}
I20260620 20:17:27.629200 11353 leader_election.cc:288] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ce5fdfca8704a4b8d7cf973a80d0a17 (127.24.20.130:44165), 26dbc9e46455412ba550ef7b0b1ce66f (127.24.20.129:37647)
I20260620 20:17:27.629642 11284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a98e8cb108443a692879da29ff7a49b" candidate_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b4a9de87c34374a22f92d632218d41" is_pre_election: true
I20260620 20:17:27.629675 11021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a98e8cb108443a692879da29ff7a49b" candidate_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" is_pre_election: true
I20260620 20:17:27.629844 11284 raft_consensus.cc:2454] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce5fdfca8704a4b8d7cf973a80d0a17 in term 0.
I20260620 20:17:27.629987 11021 raft_consensus.cc:2454] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce5fdfca8704a4b8d7cf973a80d0a17 in term 0.
I20260620 20:17:27.630283 11087 leader_election.cc:302] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b4a9de87c34374a22f92d632218d41, 8ce5fdfca8704a4b8d7cf973a80d0a17; no voters:
I20260620 20:17:27.630565 11355 raft_consensus.cc:2781] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:17:27.630645 11355 raft_consensus.cc:482] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:17:27.630685 11355 raft_consensus.cc:3037] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:27.631790 11355 raft_consensus.cc:504] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b4a9de87c34374a22f92d632218d41" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33065 } } peers { permanent_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44165 } } peers { permanent_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37647 } }
I20260620 20:17:27.632120 11355 leader_election.cc:288] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [CANDIDATE]: Term 1 election: Requested vote from peers 10b4a9de87c34374a22f92d632218d41 (127.24.20.131:33065), 26dbc9e46455412ba550ef7b0b1ce66f (127.24.20.129:37647)
I20260620 20:17:27.632573 11021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a98e8cb108443a692879da29ff7a49b" candidate_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26dbc9e46455412ba550ef7b0b1ce66f"
I20260620 20:17:27.632707 11021 raft_consensus.cc:3037] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:27.632850 11284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a98e8cb108443a692879da29ff7a49b" candidate_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b4a9de87c34374a22f92d632218d41"
I20260620 20:17:27.632956 11284 raft_consensus.cc:3037] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:27.633793 11021 raft_consensus.cc:2454] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce5fdfca8704a4b8d7cf973a80d0a17 in term 1.
I20260620 20:17:27.633926 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.628816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:37647 (local address 127.24.20.131:46681)
0620 20:17:27.629486 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:27.629501 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:27.629637 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:27.629993 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:27.629998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:27.630018 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:27.630148 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.630155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.631219 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.631223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.632488 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.632501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.633453 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.633460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.633557 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.633581 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:17:27.633668 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 20:17:27.633753 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":1,"thread_start_us":132,"threads_started":1}
I20260620 20:17:27.634158 11086 leader_election.cc:302] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26dbc9e46455412ba550ef7b0b1ce66f, 8ce5fdfca8704a4b8d7cf973a80d0a17; no voters:
I20260620 20:17:27.634367 11355 raft_consensus.cc:2781] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:27.634408 11284 raft_consensus.cc:2454] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce5fdfca8704a4b8d7cf973a80d0a17 in term 1.
I20260620 20:17:27.634478 11345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.629197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:46681 (local address 127.24.20.129:37647)
0620 20:17:27.629299 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:27.629303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:27.629572 (+ 269us) server_negotiation.cc:407] Connection header received
0620 20:17:27.629755 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:27.629759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:27.629800 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:27.629861 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:27.630283 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.631081 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.632633 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.633301 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.633433 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.633480 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:27.633696 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 20:17:27.633767 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:17:27.634608 11355 raft_consensus.cc:686] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [term 1 LEADER]: Becoming Leader. State: Replica: 8ce5fdfca8704a4b8d7cf973a80d0a17, State: Running, Role: LEADER
I20260620 20:17:27.634940 11021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a98e8cb108443a692879da29ff7a49b" candidate_uuid: "10b4a9de87c34374a22f92d632218d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" is_pre_election: true
I20260620 20:17:27.634949 11355 consensus_queue.cc:237] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b4a9de87c34374a22f92d632218d41" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33065 } } peers { permanent_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44165 } } peers { permanent_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37647 } }
I20260620 20:17:27.635115 11021 raft_consensus.cc:2379] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10b4a9de87c34374a22f92d632218d41 in current term 1: Already voted for candidate 8ce5fdfca8704a4b8d7cf973a80d0a17 in this term.
I20260620 20:17:27.635802 11332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.628352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:44165 (local address 127.24.20.131:60339)
0620 20:17:27.628486 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:27.628500 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:27.628574 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:27.629525 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:27.629528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:27.629547 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:27.629645 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.629653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.631400 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.631404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.632652 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:27.632664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.635427 (+ 2763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.635431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:27.635527 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.635549 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:17:27.635625 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 20:17:27.635698 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":47}
I20260620 20:17:27.636108 11346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.628816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:60339 (local address 127.24.20.130:44165)
0620 20:17:27.628944 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:27.628952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:27.628970 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:17:27.629016 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:27.629019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:27.629065 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:27.629150 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:27.630537 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.631284 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.634714 (+ 3430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.635299 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.635458 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.635510 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:27.635936 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 20:17:27.636004 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:17:27.636565 11152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a98e8cb108443a692879da29ff7a49b" candidate_uuid: "10b4a9de87c34374a22f92d632218d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" is_pre_election: true
I20260620 20:17:27.637105 11218 leader_election.cc:302] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10b4a9de87c34374a22f92d632218d41; no voters: 26dbc9e46455412ba550ef7b0b1ce66f, 8ce5fdfca8704a4b8d7cf973a80d0a17
I20260620 20:17:27.637337 11353 raft_consensus.cc:2726] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 20:17:27.638527 10877 catalog_manager.cc:5495] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ce5fdfca8704a4b8d7cf973a80d0a17 (127.24.20.130). New cstate: current_term: 1 leader_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b4a9de87c34374a22f92d632218d41" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ce5fdfca8704a4b8d7cf973a80d0a17" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37647 } health_report { overall_health: UNKNOWN } } }
I20260620 20:17:27.663928 11346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.660541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60808 (local address 127.24.20.130:44165)
0620 20:17:27.660686 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:27.660690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:27.660703 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:17:27.660853 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:27.660855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:27.660908 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:27.660995 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:17:27.661204 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.661697 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.662526 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:27.662748 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:27.662826 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:27.663138 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:17:27.663528 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:17:27.663645 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:27.663771 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 20:17:27.663800 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:17:27.669507 11021 raft_consensus.cc:1261] T 8a98e8cb108443a692879da29ff7a49b P 26dbc9e46455412ba550ef7b0b1ce66f [term 1 FOLLOWER]: Refusing update from remote peer 8ce5fdfca8704a4b8d7cf973a80d0a17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:27.669543 11284 raft_consensus.cc:1261] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [term 1 FOLLOWER]: Refusing update from remote peer 8ce5fdfca8704a4b8d7cf973a80d0a17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:27.670190 11355 consensus_queue.cc:1035] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b4a9de87c34374a22f92d632218d41" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 33065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:27.670449 11358 consensus_queue.cc:1035] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26dbc9e46455412ba550ef7b0b1ce66f" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 37647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:27.680327 11362 mvcc.cc:205] Tried to move back new op lower bound from 7299017308847325184 to 7299017308715683840. Current Snapshot: MvccSnapshot[applied={T|T < 7299017308847325184}]
I20260620 20:17:27.680644 11363 mvcc.cc:205] Tried to move back new op lower bound from 7299017308847325184 to 7299017308715683840. Current Snapshot: MvccSnapshot[applied={T|T < 7299017308847325184 or (T in {7299017308847325184})}]
I20260620 20:17:27.685283 10877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:27.685441 10877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:27.687675 10877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f46c6635aa04ed5968e3d24d2dea4c3: Sending DeleteTablet for 3 replicas of tablet 8a98e8cb108443a692879da29ff7a49b
I20260620 20:17:27.688493 11001 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a98e8cb108443a692879da29ff7a49b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:27 UTC) from {username='slave'} at 127.0.0.1:39348
I20260620 20:17:27.688661 11132 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a98e8cb108443a692879da29ff7a49b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:27 UTC) from {username='slave'} at 127.0.0.1:60802
I20260620 20:17:27.688949 11264 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a98e8cb108443a692879da29ff7a49b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:27 UTC) from {username='slave'} at 127.0.0.1:55708
I20260620 20:17:27.689136 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 10939
I20260620 20:17:27.689396 11371 tablet_replica.cc:330] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17: stopping tablet replica
I20260620 20:17:27.690364 11371 raft_consensus.cc:2229] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:17:27.690673 11371 raft_consensus.cc:2258] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:27.692096 11371 ts_tablet_manager.cc:1905] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:17:27.694424 11372 tablet_replica.cc:330] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41: stopping tablet replica
I20260620 20:17:27.694774 11372 raft_consensus.cc:2229] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:17:27.694808 11371 ts_tablet_manager.cc:1918] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:27.694904 11371 log.cc:1196] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/8a98e8cb108443a692879da29ff7a49b
I20260620 20:17:27.695200 11371 ts_tablet_manager.cc:1939] T 8a98e8cb108443a692879da29ff7a49b P 8ce5fdfca8704a4b8d7cf973a80d0a17: Deleting consensus metadata
I20260620 20:17:27.695307 11372 raft_consensus.cc:2258] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 20:17:27.695819 10864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:37647 (error 108)
I20260620 20:17:27.696327 10865 catalog_manager.cc:4841] TS 8ce5fdfca8704a4b8d7cf973a80d0a17 (127.24.20.130:44165): tablet 8a98e8cb108443a692879da29ff7a49b (table test-table [id=36b45e8b707d4280aa0303e8e81a090a]) successfully deleted
I20260620 20:17:27.696970 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 11070
I20260620 20:17:27.697314 11372 ts_tablet_manager.cc:1905] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:17:27.697717 11344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:27.696924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:37647 (local address 127.0.0.1:39364)
0620 20:17:27.697521 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:27.697605 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:37647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502}
W20260620 20:17:27.698880 10864 catalog_manager.cc:4584] TS 26dbc9e46455412ba550ef7b0b1ce66f (127.24.20.129:37647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a98e8cb108443a692879da29ff7a49b: Network error: Client connection negotiation failed: client connection to 127.24.20.129:37647: connect: Connection refused (error 111)
I20260620 20:17:27.700488 11372 ts_tablet_manager.cc:1918] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:27.700615 11372 log.cc:1196] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/8a98e8cb108443a692879da29ff7a49b
I20260620 20:17:27.700932 11372 ts_tablet_manager.cc:1939] T 8a98e8cb108443a692879da29ff7a49b P 10b4a9de87c34374a22f92d632218d41: Deleting consensus metadata
I20260620 20:17:27.702068 10865 catalog_manager.cc:4841] TS 10b4a9de87c34374a22f92d632218d41 (127.24.20.131:33065): tablet 8a98e8cb108443a692879da29ff7a49b (table test-table [id=36b45e8b707d4280aa0303e8e81a090a]) successfully deleted
I20260620 20:17:27.703668 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 11201
I20260620 20:17:27.710688 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 10848
2026-06-20T20:17:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1028 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:27 dist-test-slave-pflr krb5kdc[11376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:27 dist-test-slave-pflr krb5kdc[11376](info): set up 2 sockets
Jun 20 20:17:27 dist-test-slave-pflr krb5kdc[11376](info): commencing operation
krb5kdc: starting...
W20260620 20:17:28.744931 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:28 dist-test-slave-pflr krb5kdc[11376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:17:28.861793 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:42785
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:41259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:42785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:28.967473 11389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:28.967764 11389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:28.967845 11389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:28.971814 11389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:28.971896 11389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:28.971947 11389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:28.972012 11389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:28.972049 11389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:28.977140 11389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:41259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:42785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:42785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:28.978102 11389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:28.979080 11389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:17:28.987596 11389 server_base.cc:1034] running on GCE node
W20260620 20:17:28.987545 11395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:28.987536 11397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:28.987571 11394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:28.988287 11389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:28.989418 11389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:28.990630 11389 hybrid_clock.cc:648] HybridClock initialized: now 1781986648990606 us; error 43 us; skew 500 ppm
Jun 20 20:17:28 dist-test-slave-pflr krb5kdc[11376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986648, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:28.993279 11389 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:17:28.994179 11389 webserver.cc:460] Webserver started at http://127.24.20.190:43337/ using document root <none> and password file <none>
I20260620 20:17:28.994712 11389 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:28.994799 11389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:28.995020 11389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:28.997023 11389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "edb4224e9beb43c2ab6c8dc0bff8aa00"
format_stamp: "Formatted at 2026-06-20 20:17:28 on dist-test-slave-pflr"
server_key: "0bbac4f67c61c71164da835785eda50f"
server_key_iv: "4b8758b5cd8697164da396cab21a695f"
server_key_version: "encryptionkey@0"
I20260620 20:17:28.997568 11389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "edb4224e9beb43c2ab6c8dc0bff8aa00"
format_stamp: "Formatted at 2026-06-20 20:17:28 on dist-test-slave-pflr"
server_key: "0bbac4f67c61c71164da835785eda50f"
server_key_iv: "4b8758b5cd8697164da396cab21a695f"
server_key_version: "encryptionkey@0"
I20260620 20:17:29.001055 11389 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:29.003330 11404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:29.004499 11389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:17:29.004666 11389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "edb4224e9beb43c2ab6c8dc0bff8aa00"
format_stamp: "Formatted at 2026-06-20 20:17:28 on dist-test-slave-pflr"
server_key: "0bbac4f67c61c71164da835785eda50f"
server_key_iv: "4b8758b5cd8697164da396cab21a695f"
server_key_version: "encryptionkey@0"
I20260620 20:17:29.004781 11389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:29.032876 11389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:29.035674 11389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:29.035879 11389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:29.042554 11389 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:42785
I20260620 20:17:29.042583 11456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:42785 every 8 connection(s)
I20260620 20:17:29.043475 11389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:29.046111 11457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:29.047530 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 11389
I20260620 20:17:29.047668 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:29.047976 24658 external_mini_cluster.cc:1428] Setting key 2190eedc564bed3b4ef0a97dafc78f25
I20260620 20:17:29.051968 11457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00: Bootstrap starting.
I20260620 20:17:29.054083 11457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:29.054973 11457 log.cc:824] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:17:29 dist-test-slave-pflr krb5kdc[11376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:17:29.056886 11457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00: No bootstrap required, opened a new log
I20260620 20:17:29.059581 11457 raft_consensus.cc:348] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb4224e9beb43c2ab6c8dc0bff8aa00" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42785 } }
I20260620 20:17:29.059803 11457 raft_consensus.cc:374] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:29.059880 11457 raft_consensus.cc:729] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb4224e9beb43c2ab6c8dc0bff8aa00, State: Initialized, Role: FOLLOWER
I20260620 20:17:29.060328 11457 consensus_queue.cc:260] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb4224e9beb43c2ab6c8dc0bff8aa00" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42785 } }
I20260620 20:17:29.060474 11457 raft_consensus.cc:388] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:29.060565 11457 raft_consensus.cc:482] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:29.060686 11457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:29.061600 11457 raft_consensus.cc:504] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb4224e9beb43c2ab6c8dc0bff8aa00" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42785 } }
I20260620 20:17:29.061931 11457 leader_election.cc:302] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edb4224e9beb43c2ab6c8dc0bff8aa00; no voters:
I20260620 20:17:29.062203 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.049346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.24.20.190:42785)
0620 20:17:29.049747 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:29.049761 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:29.049795 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:17:29.050396 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:29.050413 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:29.050723 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:29.051046 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:29.051813 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.052853 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.053714 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.054166 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.054328 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.057092 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:29.057097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:29.057109 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:29.057133 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:29.059259 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:29.059709 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:29.059714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:29.059719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:29.059850 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:29.060172 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:29.060176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:29.060177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:29.060630 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:29.060825 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:29.061077 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 20:17:29.061351 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20260620 20:17:29.062305 11462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:29.062233 11457 leader_election.cc:288] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:29.062480 11462 raft_consensus.cc:686] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [term 1 LEADER]: Becoming Leader. State: Replica: edb4224e9beb43c2ab6c8dc0bff8aa00, State: Running, Role: LEADER
I20260620 20:17:29.062749 11462 consensus_queue.cc:237] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb4224e9beb43c2ab6c8dc0bff8aa00" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42785 } }
I20260620 20:17:29.063576 11457 sys_catalog.cc:564] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:29.063858 11462 sys_catalog.cc:455] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edb4224e9beb43c2ab6c8dc0bff8aa00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb4224e9beb43c2ab6c8dc0bff8aa00" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42785 } } }
I20260620 20:17:29.063975 11462 sys_catalog.cc:458] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:29.064378 11464 sys_catalog.cc:455] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edb4224e9beb43c2ab6c8dc0bff8aa00. Latest consensus state: current_term: 1 leader_uuid: "edb4224e9beb43c2ab6c8dc0bff8aa00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb4224e9beb43c2ab6c8dc0bff8aa00" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42785 } } }
I20260620 20:17:29.064538 11464 sys_catalog.cc:458] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00 [sys.catalog]: This master's current role is: LEADER
W20260620 20:17:29.068086 11477 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:17:29.068176 11477 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:17:29.068243 11466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:29.069010 11466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:29.073751 11466 catalog_manager.cc:1347] Generated new cluster ID: 94b55a4e05b34addb57fa99e11d3ac5e
I20260620 20:17:29.073809 11466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:29.105043 11466 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:29.106333 11466 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:29.114925 11466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00: Generated new TSK 0
I20260620 20:17:29.115685 11466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.20.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
Entry for principal kudu/127.24.20.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab.
I20260620 20:17:29.151744 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42785
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:41259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:29.259246 11483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:29.259475 11483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:29.259516 11483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:29.263507 11483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:29.263557 11483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:29.263665 11483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:17:29.268970 11483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:41259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.129.keytab
--principal=kudu/127.24.20.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:29.269897 11483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:29.270861 11483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:29.280067 11488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:29.280035 11489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:29.280114 11483 server_base.cc:1034] running on GCE node
W20260620 20:17:29.280059 11491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:29.280668 11483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:29.281263 11483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:29.282438 11483 hybrid_clock.cc:648] HybridClock initialized: now 1781986649282426 us; error 24 us; skew 500 ppm
Jun 20 20:17:29 dist-test-slave-pflr krb5kdc[11376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986649, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:29.284983 11483 init.cc:377] Logged in from keytab as kudu/127.24.20.129@KRBTEST.COM (short username kudu)
I20260620 20:17:29.285734 11483 webserver.cc:460] Webserver started at http://127.24.20.129:40073/ using document root <none> and password file <none>
I20260620 20:17:29.286245 11483 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:29.286329 11483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:29.286537 11483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:29.288579 11483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "34a7bc372b424ff68d7ac4f0d8e070e3"
format_stamp: "Formatted at 2026-06-20 20:17:29 on dist-test-slave-pflr"
server_key: "8907b1ebb828ef039883c369f0061a3b"
server_key_iv: "3c3604c4ea550b132d4e3a24778a5735"
server_key_version: "encryptionkey@0"
I20260620 20:17:29.289148 11483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "34a7bc372b424ff68d7ac4f0d8e070e3"
format_stamp: "Formatted at 2026-06-20 20:17:29 on dist-test-slave-pflr"
server_key: "8907b1ebb828ef039883c369f0061a3b"
server_key_iv: "3c3604c4ea550b132d4e3a24778a5735"
server_key_version: "encryptionkey@0"
I20260620 20:17:29.292469 11483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:17:29.294814 11498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:29.296084 11483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:29.296200 11483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "34a7bc372b424ff68d7ac4f0d8e070e3"
format_stamp: "Formatted at 2026-06-20 20:17:29 on dist-test-slave-pflr"
server_key: "8907b1ebb828ef039883c369f0061a3b"
server_key_iv: "3c3604c4ea550b132d4e3a24778a5735"
server_key_version: "encryptionkey@0"
I20260620 20:17:29.296298 11483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:29.324694 11483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:29.327598 11483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:29.327796 11483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:29.328379 11483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:29.329289 11483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:29.329335 11483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:29.329407 11483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:29.329437 11483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:29.338141 11483 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:34555
I20260620 20:17:29.338204 11611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:34555 every 8 connection(s)
I20260620 20:17:29.339038 11483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:17:29.347556 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 11483
I20260620 20:17:29.347702 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:17:29.348024 24658 external_mini_cluster.cc:1428] Setting key a32d9bc19202c529b2a9e943da2c3011
Jun 20 20:17:29 dist-test-slave-pflr krb5kdc[11376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986649, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.129@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:17:29.352684 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.340792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:39149 (local address 127.24.20.190:42785)
0620 20:17:29.340932 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:29.340935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:29.341703 (+ 768us) server_negotiation.cc:407] Connection header received
0620 20:17:29.342969 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:29.342972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:29.343030 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:29.343149 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:29.344230 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.344841 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.345942 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.346180 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.346292 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.348865 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:29.348868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:29.348870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:29.348894 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:29.350610 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:29.351069 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:29.351074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:29.351076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:29.351174 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:29.351576 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:29.351579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:29.351580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:29.351746 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:29.351821 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:29.352403 (+ 582us) server_negotiation.cc:299] Negotiation successful
0620 20:17:29.352503 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:17:29.353781 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.341135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42785 (local address 127.24.20.129:39149)
0620 20:17:29.341569 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:29.341608 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:29.342771 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:29.343311 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:29.343317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:29.343654 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:29.344092 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.344105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.344958 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.344962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.345812 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.345821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.346322 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.346326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.346492 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.346996 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:29.347012 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:29.348676 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:29.350725 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:29.350729 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:29.350739 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:29.350973 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:29.351280 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:29.351284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:29.351286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:29.351475 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:29.351845 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:29.351852 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:29.352155 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 20:17:29.352371 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 20:17:29.352741 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":114,"threads_started":1}
I20260620 20:17:29.355022 11612 heartbeater.cc:347] Connected to a master server at 127.24.20.190:42785
I20260620 20:17:29.355257 11612 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:29.355747 11612 heartbeater.cc:511] Master 127.24.20.190:42785 requested a full tablet report, sending...
I20260620 20:17:29.357218 11421 ts_manager.cc:194] Registered new tserver with Master: 34a7bc372b424ff68d7ac4f0d8e070e3 (127.24.20.129:34555)
WARNING: no policy specified for kudu/127.24.20.130@KRBTEST.COM; defaulting to no policy
I20260620 20:17:29.358372 11421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.129@KRBTEST.COM'} at 127.24.20.129:39149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
Entry for principal kudu/127.24.20.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab.
I20260620 20:17:29.375602 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42785
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:41259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:29.483018 11617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:29.483250 11617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:29.483317 11617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:29.487301 11617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:29.487350 11617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:29.487406 11617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:17:29.492444 11617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:41259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.130.keytab
--principal=kudu/127.24.20.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:29.493412 11617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:29.494361 11617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:29.503610 11623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:29.503633 11622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:29.503808 11617 server_base.cc:1034] running on GCE node
W20260620 20:17:29.503610 11625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:29.504315 11617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:29.505057 11617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:29.506238 11617 hybrid_clock.cc:648] HybridClock initialized: now 1781986649506220 us; error 33 us; skew 500 ppm
Jun 20 20:17:29 dist-test-slave-pflr krb5kdc[11376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986649, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:29.509050 11617 init.cc:377] Logged in from keytab as kudu/127.24.20.130@KRBTEST.COM (short username kudu)
I20260620 20:17:29.509922 11617 webserver.cc:460] Webserver started at http://127.24.20.130:34661/ using document root <none> and password file <none>
I20260620 20:17:29.510471 11617 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:29.510558 11617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:29.510798 11617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:29.512857 11617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "c8b3fd3b22ee45328853311ab94686e7"
format_stamp: "Formatted at 2026-06-20 20:17:29 on dist-test-slave-pflr"
server_key: "b6faa716cd922cb000af8921f52714c1"
server_key_iv: "6fe3f789c6345cd603b816fb7823447f"
server_key_version: "encryptionkey@0"
I20260620 20:17:29.513403 11617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "c8b3fd3b22ee45328853311ab94686e7"
format_stamp: "Formatted at 2026-06-20 20:17:29 on dist-test-slave-pflr"
server_key: "b6faa716cd922cb000af8921f52714c1"
server_key_iv: "6fe3f789c6345cd603b816fb7823447f"
server_key_version: "encryptionkey@0"
I20260620 20:17:29.517062 11617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:29.519269 11632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:29.520427 11617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:29.520563 11617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "c8b3fd3b22ee45328853311ab94686e7"
format_stamp: "Formatted at 2026-06-20 20:17:29 on dist-test-slave-pflr"
server_key: "b6faa716cd922cb000af8921f52714c1"
server_key_iv: "6fe3f789c6345cd603b816fb7823447f"
server_key_version: "encryptionkey@0"
I20260620 20:17:29.520723 11617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:29.539716 11617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:29.543087 11617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:29.543282 11617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:29.543844 11617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:29.544670 11617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:29.544740 11617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:29.544806 11617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:29.544858 11617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:29.553576 11617 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:41825
I20260620 20:17:29.553617 11745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:41825 every 8 connection(s)
I20260620 20:17:29.554412 11617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:17:29.561523 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 11617
I20260620 20:17:29.561684 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:17:29.562031 24658 external_mini_cluster.cc:1428] Setting key 9cd08d3ce7b8069a2a85a30bdf0d3eeb
Jun 20 20:17:29 dist-test-slave-pflr krb5kdc[11376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986649, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.130@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:17:29.568638 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.556020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:35425 (local address 127.24.20.190:42785)
0620 20:17:29.556163 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:29.556167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:29.556973 (+ 806us) server_negotiation.cc:407] Connection header received
0620 20:17:29.558252 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:29.558255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:29.558316 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:29.558393 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:29.559497 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.560036 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.561185 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.561531 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.561667 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.564624 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:29.564629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:29.564632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:29.564658 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:29.566445 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:29.566928 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:29.566934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:29.566939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:29.567031 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:29.567333 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:29.567339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:29.567343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:29.567608 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:29.567717 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:29.568223 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 20:17:29.568350 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:17:29.569233 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.556368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42785 (local address 127.24.20.130:35425)
0620 20:17:29.556826 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:29.556867 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:29.558029 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:29.558554 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:29.558560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:29.558886 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:29.559343 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.559356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.560161 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.560165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.561051 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.561060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.561777 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.561783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.562028 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.562683 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:29.562703 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:29.564417 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:29.566575 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:29.566580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:29.566590 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:29.566828 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:29.567124 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:29.567126 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:29.567127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:29.567245 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:29.567708 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:29.567712 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:29.567980 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 20:17:29.568136 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 20:17:29.568356 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":105,"threads_started":1}
I20260620 20:17:29.570279 11746 heartbeater.cc:347] Connected to a master server at 127.24.20.190:42785
I20260620 20:17:29.570521 11746 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:29.570967 11746 heartbeater.cc:511] Master 127.24.20.190:42785 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.20.131@KRBTEST.COM; defaulting to no policy
I20260620 20:17:29.571924 11421 ts_manager.cc:194] Registered new tserver with Master: c8b3fd3b22ee45328853311ab94686e7 (127.24.20.130:41825)
I20260620 20:17:29.572515 11421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.130@KRBTEST.COM'} at 127.24.20.130:35425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
Entry for principal kudu/127.24.20.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab.
I20260620 20:17:29.589068 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42785
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:41259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:29.695798 11751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:29.696034 11751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:29.696085 11751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:29.700044 11751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:29.700098 11751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:29.700155 11751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:17:29.705219 11751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:41259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.131.keytab
--principal=kudu/127.24.20.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:29.706081 11751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:29.706945 11751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:29.716038 11756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:29.716091 11757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:29.716013 11759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:29.716214 11751 server_base.cc:1034] running on GCE node
I20260620 20:17:29.716697 11751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:29.717296 11751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:29.718472 11751 hybrid_clock.cc:648] HybridClock initialized: now 1781986649718450 us; error 32 us; skew 500 ppm
Jun 20 20:17:29 dist-test-slave-pflr krb5kdc[11376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986649, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:29.721127 11751 init.cc:377] Logged in from keytab as kudu/127.24.20.131@KRBTEST.COM (short username kudu)
I20260620 20:17:29.721904 11751 webserver.cc:460] Webserver started at http://127.24.20.131:34301/ using document root <none> and password file <none>
I20260620 20:17:29.722460 11751 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:29.722551 11751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:29.722785 11751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:29.724927 11751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "30537e7ce6194154a8c11f558c3befb5"
format_stamp: "Formatted at 2026-06-20 20:17:29 on dist-test-slave-pflr"
server_key: "3efddcd22e8507a6d787f899d4e0da7e"
server_key_iv: "c7cdce87c43cb3fd5e3dc9d6c1091413"
server_key_version: "encryptionkey@0"
I20260620 20:17:29.725464 11751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "30537e7ce6194154a8c11f558c3befb5"
format_stamp: "Formatted at 2026-06-20 20:17:29 on dist-test-slave-pflr"
server_key: "3efddcd22e8507a6d787f899d4e0da7e"
server_key_iv: "c7cdce87c43cb3fd5e3dc9d6c1091413"
server_key_version: "encryptionkey@0"
I20260620 20:17:29.728926 11751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:29.731114 11766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:29.732229 11751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:29.732362 11751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "30537e7ce6194154a8c11f558c3befb5"
format_stamp: "Formatted at 2026-06-20 20:17:29 on dist-test-slave-pflr"
server_key: "3efddcd22e8507a6d787f899d4e0da7e"
server_key_iv: "c7cdce87c43cb3fd5e3dc9d6c1091413"
server_key_version: "encryptionkey@0"
I20260620 20:17:29.732465 11751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:29.744300 11751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:29.747399 11751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:29.747540 11751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:29.748013 11751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:29.748760 11751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:29.748803 11751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:29.748842 11751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:29.748869 11751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:29.757292 11751 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:36813
I20260620 20:17:29.757315 11879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:36813 every 8 connection(s)
I20260620 20:17:29.758229 11751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:17:29.764629 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 11751
I20260620 20:17:29.764782 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:17:29.765113 24658 external_mini_cluster.cc:1428] Setting key 14d7f6f804af2d8cfdadd2b3fecaf054
Jun 20 20:17:29 dist-test-slave-pflr krb5kdc[11376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986649, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.131@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:17:29.771692 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.759997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:57583 (local address 127.24.20.190:42785)
0620 20:17:29.760164 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:29.760169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:29.760958 (+ 789us) server_negotiation.cc:407] Connection header received
0620 20:17:29.762170 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:29.762172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:29.762218 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:29.762292 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:29.763443 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.763972 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.765544 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.765870 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.765969 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.768566 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:29.768569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:29.768571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:29.768635 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:29.770071 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:29.770486 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:29.770488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:29.770489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:29.770566 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:29.770836 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:29.770838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:29.770839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:29.770994 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:29.771067 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:29.771481 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 20:17:29.771577 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:17:29.772454 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.760349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:42785 (local address 127.24.20.131:57583)
0620 20:17:29.760819 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:29.760859 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:29.761976 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:29.762463 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:29.762470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:29.762794 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:29.763264 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.763276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.764081 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.764084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.765429 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.765439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.766049 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.766056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.766229 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.766847 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:29.766868 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:29.768418 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:29.770163 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:29.770168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:29.770178 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:29.770405 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:29.770642 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:29.770644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:29.770645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:29.770762 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:29.771084 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:29.771088 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:29.771292 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 20:17:29.771439 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 20:17:29.771668 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":106,"threads_started":1}
I20260620 20:17:29.773519 11880 heartbeater.cc:347] Connected to a master server at 127.24.20.190:42785
I20260620 20:17:29.773743 11880 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:29.774189 11880 heartbeater.cc:511] Master 127.24.20.190:42785 requested a full tablet report, sending...
I20260620 20:17:29.775121 11421 ts_manager.cc:194] Registered new tserver with Master: 30537e7ce6194154a8c11f558c3befb5 (127.24.20.131:36813)
I20260620 20:17:29.775928 11421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.20.131@KRBTEST.COM'} at 127.24.20.131:57583
I20260620 20:17:29.778842 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:17:29.788117 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.780712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.24.20.190:42785)
0620 20:17:29.780842 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:29.780848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:29.781052 (+ 204us) server_negotiation.cc:407] Connection header received
0620 20:17:29.781205 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:29.781209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:29.781255 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:29.781336 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:29.782006 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.782514 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.783353 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.783576 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.783683 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.785085 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:29.785089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:29.785091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:29.785114 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:29.786501 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:29.786927 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:29.786931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:29.786932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:29.787000 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:29.787266 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:29.787270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:29.787271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:29.787470 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:29.787616 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:29.787848 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 20:17:29.787955 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:17:29.790799 11421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:17:29.793871 11421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:17:29.808115 11891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.801719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:34555 (local address 127.0.0.1:56896)
0620 20:17:29.802264 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:29.802311 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:29.802447 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:29.803586 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:29.803589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:29.803604 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:29.803788 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.803797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.805168 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.805174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.806257 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.806266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.807403 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.807408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.807623 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.807682 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 20:17:29.807785 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:17:29.807946 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":121,"thread_start_us":64,"threads_started":1}
I20260620 20:17:29.808115 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.801788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:36813 (local address 127.0.0.1:50654)
0620 20:17:29.802433 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:29.802447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:29.802534 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:29.803414 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:29.803420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:29.803442 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:29.803582 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.803589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.805074 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.805077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.806432 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.806443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.807424 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.807427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.807618 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.807674 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 20:17:29.807785 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:17:29.807943 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":205,"thread_start_us":94,"threads_started":1}
I20260620 20:17:29.808259 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.801812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.24.20.129:34555)
0620 20:17:29.803176 (+ 1364us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:29.803190 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:29.803208 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:17:29.803274 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:29.803279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:29.803433 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:29.803745 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:29.803882 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.804980 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.806379 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.807221 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.807337 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.807492 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:29.808035 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 20:17:29.808124 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1274,"thread_start_us":105,"threads_started":1}
I20260620 20:17:29.808514 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.802500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50654 (local address 127.24.20.131:36813)
0620 20:17:29.802949 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:29.802961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:29.802985 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:17:29.803063 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:29.803068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:29.803275 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:29.803424 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:29.803706 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.804932 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.806552 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.807322 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.807410 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.807516 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:29.807832 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 20:17:29.807927 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":151,"threads_started":1}
I20260620 20:17:29.809190 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.802081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:41825 (local address 127.0.0.1:33754)
0620 20:17:29.802861 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:29.802874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:29.802954 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:29.803568 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:29.803570 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:29.803585 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:29.803728 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.803737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.804770 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.804775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.806432 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.806443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.808809 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.808813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.808899 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.808922 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:17:29.808992 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:17:29.809057 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":151,"thread_start_us":91,"threads_started":1}
I20260620 20:17:29.809585 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.802497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.24.20.130:41825)
0620 20:17:29.802951 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:29.802964 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:29.802988 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:17:29.803075 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:29.803080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:29.803280 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:29.803429 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:29.803851 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.804576 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.807564 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.808445 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.808571 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.808745 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:29.809361 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 20:17:29.809455 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":144,"threads_started":1}
I20260620 20:17:29.811301 11814 tablet_service.cc:1467] Processing CreateTablet for tablet 1700c47495dd45a783dad70ed5aac6bf (DEFAULT_TABLE table=test-table [id=9ab6be18dd6f47589d33f492d80a7da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:29.811437 11546 tablet_service.cc:1467] Processing CreateTablet for tablet 1700c47495dd45a783dad70ed5aac6bf (DEFAULT_TABLE table=test-table [id=9ab6be18dd6f47589d33f492d80a7da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:29.812315 11814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1700c47495dd45a783dad70ed5aac6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:29.811961 11680 tablet_service.cc:1467] Processing CreateTablet for tablet 1700c47495dd45a783dad70ed5aac6bf (DEFAULT_TABLE table=test-table [id=9ab6be18dd6f47589d33f492d80a7da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:29.812462 11546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1700c47495dd45a783dad70ed5aac6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:29.812861 11680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1700c47495dd45a783dad70ed5aac6bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:29.818287 11897 tablet_bootstrap.cc:492] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5: Bootstrap starting.
I20260620 20:17:29.819873 11898 tablet_bootstrap.cc:492] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7: Bootstrap starting.
I20260620 20:17:29.820492 11897 tablet_bootstrap.cc:654] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:29.821543 11898 tablet_bootstrap.cc:654] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:29.821532 11897 log.cc:824] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:29.822482 11898 log.cc:824] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:29.823154 11899 tablet_bootstrap.cc:492] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3: Bootstrap starting.
I20260620 20:17:29.823436 11897 tablet_bootstrap.cc:492] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5: No bootstrap required, opened a new log
I20260620 20:17:29.823639 11897 ts_tablet_manager.cc:1397] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:17:29.824909 11899 tablet_bootstrap.cc:654] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:29.825934 11898 tablet_bootstrap.cc:492] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7: No bootstrap required, opened a new log
I20260620 20:17:29.826007 11899 log.cc:824] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:29.826128 11898 ts_tablet_manager.cc:1397] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 20:17:29.826889 11897 raft_consensus.cc:348] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a7bc372b424ff68d7ac4f0d8e070e3" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34555 } } peers { permanent_uuid: "30537e7ce6194154a8c11f558c3befb5" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 36813 } } peers { permanent_uuid: "c8b3fd3b22ee45328853311ab94686e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 41825 } }
I20260620 20:17:29.827123 11897 raft_consensus.cc:374] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:29.827183 11897 raft_consensus.cc:729] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30537e7ce6194154a8c11f558c3befb5, State: Initialized, Role: FOLLOWER
I20260620 20:17:29.827618 11897 consensus_queue.cc:260] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a7bc372b424ff68d7ac4f0d8e070e3" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34555 } } peers { permanent_uuid: "30537e7ce6194154a8c11f558c3befb5" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 36813 } } peers { permanent_uuid: "c8b3fd3b22ee45328853311ab94686e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 41825 } }
I20260620 20:17:29.827841 11899 tablet_bootstrap.cc:492] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3: No bootstrap required, opened a new log
I20260620 20:17:29.828011 11899 ts_tablet_manager.cc:1397] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 20:17:29.828485 11897 ts_tablet_manager.cc:1428] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:17:29.828866 11880 heartbeater.cc:503] Master 127.24.20.190:42785 was elected leader, sending a full tablet report...
I20260620 20:17:29.829898 11898 raft_consensus.cc:348] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a7bc372b424ff68d7ac4f0d8e070e3" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34555 } } peers { permanent_uuid: "30537e7ce6194154a8c11f558c3befb5" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 36813 } } peers { permanent_uuid: "c8b3fd3b22ee45328853311ab94686e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 41825 } }
I20260620 20:17:29.830140 11898 raft_consensus.cc:374] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:29.830188 11898 raft_consensus.cc:729] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b3fd3b22ee45328853311ab94686e7, State: Initialized, Role: FOLLOWER
I20260620 20:17:29.830567 11898 consensus_queue.cc:260] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a7bc372b424ff68d7ac4f0d8e070e3" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34555 } } peers { permanent_uuid: "30537e7ce6194154a8c11f558c3befb5" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 36813 } } peers { permanent_uuid: "c8b3fd3b22ee45328853311ab94686e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 41825 } }
I20260620 20:17:29.831197 11746 heartbeater.cc:503] Master 127.24.20.190:42785 was elected leader, sending a full tablet report...
I20260620 20:17:29.831485 11898 ts_tablet_manager.cc:1428] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 20:17:29.832475 11899 raft_consensus.cc:348] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a7bc372b424ff68d7ac4f0d8e070e3" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34555 } } peers { permanent_uuid: "30537e7ce6194154a8c11f558c3befb5" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 36813 } } peers { permanent_uuid: "c8b3fd3b22ee45328853311ab94686e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 41825 } }
I20260620 20:17:29.832787 11899 raft_consensus.cc:374] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:29.832851 11899 raft_consensus.cc:729] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a7bc372b424ff68d7ac4f0d8e070e3, State: Initialized, Role: FOLLOWER
I20260620 20:17:29.833206 11899 consensus_queue.cc:260] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a7bc372b424ff68d7ac4f0d8e070e3" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34555 } } peers { permanent_uuid: "30537e7ce6194154a8c11f558c3befb5" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 36813 } } peers { permanent_uuid: "c8b3fd3b22ee45328853311ab94686e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 41825 } }
I20260620 20:17:29.833843 11612 heartbeater.cc:503] Master 127.24.20.190:42785 was elected leader, sending a full tablet report...
I20260620 20:17:29.834151 11899 ts_tablet_manager.cc:1428] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 20:17:29.840116 11613 tablet.cc:2369] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:17:29.897891 11904 raft_consensus.cc:482] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:29.898129 11904 raft_consensus.cc:504] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a7bc372b424ff68d7ac4f0d8e070e3" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34555 } } peers { permanent_uuid: "30537e7ce6194154a8c11f558c3befb5" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 36813 } } peers { permanent_uuid: "c8b3fd3b22ee45328853311ab94686e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 41825 } }
I20260620 20:17:29.899345 11904 leader_election.cc:288] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34a7bc372b424ff68d7ac4f0d8e070e3 (127.24.20.129:34555), 30537e7ce6194154a8c11f558c3befb5 (127.24.20.131:36813)
I20260620 20:17:29.902963 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.899496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:34555 (local address 127.24.20.130:45843)
0620 20:17:29.899653 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:29.899670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:29.899758 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:29.900171 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:29.900177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:29.900196 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:29.900324 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.900330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.901111 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.901115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.901887 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.901896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.902533 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.902537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.902642 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.902665 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:17:29.902762 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 20:17:29.902834 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 20:17:29.902963 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.899608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:45843 (local address 127.24.20.129:34555)
0620 20:17:29.899744 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:29.899751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:29.899769 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:17:29.899937 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:29.899944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:29.900006 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:29.900100 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:29.900436 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.900998 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.902018 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.902416 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.902544 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.902606 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:29.902778 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 20:17:29.902847 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:17:29.903345 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.899595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:36813 (local address 127.24.20.130:36089)
0620 20:17:29.899983 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:29.899994 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:29.900134 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:29.900541 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:29.900543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:29.900564 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:29.900747 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.900754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.901443 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.901449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.902226 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:29.902234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.902866 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.902870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:29.902964 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.902986 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:17:29.903122 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 20:17:29.903210 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":1,"thread_start_us":132,"threads_started":1}
I20260620 20:17:29.903345 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.899812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:36089 (local address 127.24.20.131:36813)
0620 20:17:29.899923 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:29.899928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:29.900103 (+ 175us) server_negotiation.cc:407] Connection header received
0620 20:17:29.900271 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:29.900274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:29.900328 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:29.900380 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:29.900846 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.901333 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.902358 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.902750 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.902878 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.902923 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:29.903151 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 20:17:29.903235 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 20:17:29.903594 11564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1700c47495dd45a783dad70ed5aac6bf" candidate_uuid: "c8b3fd3b22ee45328853311ab94686e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a7bc372b424ff68d7ac4f0d8e070e3" is_pre_election: true
I20260620 20:17:29.903923 11564 raft_consensus.cc:2454] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8b3fd3b22ee45328853311ab94686e7 in term 0.
I20260620 20:17:29.903960 11834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1700c47495dd45a783dad70ed5aac6bf" candidate_uuid: "c8b3fd3b22ee45328853311ab94686e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30537e7ce6194154a8c11f558c3befb5" is_pre_election: true
I20260620 20:17:29.904229 11834 raft_consensus.cc:2454] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8b3fd3b22ee45328853311ab94686e7 in term 0.
I20260620 20:17:29.904453 11633 leader_election.cc:302] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a7bc372b424ff68d7ac4f0d8e070e3, c8b3fd3b22ee45328853311ab94686e7; no voters:
I20260620 20:17:29.904757 11904 raft_consensus.cc:2781] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:17:29.904840 11904 raft_consensus.cc:482] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:17:29.904884 11904 raft_consensus.cc:3037] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:29.905893 11904 raft_consensus.cc:504] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a7bc372b424ff68d7ac4f0d8e070e3" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34555 } } peers { permanent_uuid: "30537e7ce6194154a8c11f558c3befb5" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 36813 } } peers { permanent_uuid: "c8b3fd3b22ee45328853311ab94686e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 41825 } }
I20260620 20:17:29.906255 11904 leader_election.cc:288] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [CANDIDATE]: Term 1 election: Requested vote from peers 34a7bc372b424ff68d7ac4f0d8e070e3 (127.24.20.129:34555), 30537e7ce6194154a8c11f558c3befb5 (127.24.20.131:36813)
I20260620 20:17:29.906626 11564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1700c47495dd45a783dad70ed5aac6bf" candidate_uuid: "c8b3fd3b22ee45328853311ab94686e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a7bc372b424ff68d7ac4f0d8e070e3"
I20260620 20:17:29.906749 11564 raft_consensus.cc:3037] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:29.906821 11834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1700c47495dd45a783dad70ed5aac6bf" candidate_uuid: "c8b3fd3b22ee45328853311ab94686e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30537e7ce6194154a8c11f558c3befb5"
I20260620 20:17:29.906946 11834 raft_consensus.cc:3037] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:29.907958 11564 raft_consensus.cc:2454] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8b3fd3b22ee45328853311ab94686e7 in term 1.
I20260620 20:17:29.908074 11834 raft_consensus.cc:2454] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8b3fd3b22ee45328853311ab94686e7 in term 1.
I20260620 20:17:29.908277 11633 leader_election.cc:302] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34a7bc372b424ff68d7ac4f0d8e070e3, c8b3fd3b22ee45328853311ab94686e7; no voters:
I20260620 20:17:29.908461 11904 raft_consensus.cc:2781] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:29.908746 11904 raft_consensus.cc:686] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [term 1 LEADER]: Becoming Leader. State: Replica: c8b3fd3b22ee45328853311ab94686e7, State: Running, Role: LEADER
I20260620 20:17:29.909093 11904 consensus_queue.cc:237] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a7bc372b424ff68d7ac4f0d8e070e3" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34555 } } peers { permanent_uuid: "30537e7ce6194154a8c11f558c3befb5" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 36813 } } peers { permanent_uuid: "c8b3fd3b22ee45328853311ab94686e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 41825 } }
I20260620 20:17:29.912076 11420 catalog_manager.cc:5495] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8b3fd3b22ee45328853311ab94686e7 (127.24.20.130). New cstate: current_term: 1 leader_uuid: "c8b3fd3b22ee45328853311ab94686e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34a7bc372b424ff68d7ac4f0d8e070e3" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30537e7ce6194154a8c11f558c3befb5" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 36813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8b3fd3b22ee45328853311ab94686e7" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 41825 } health_report { overall_health: HEALTHY } } }
I20260620 20:17:29.927357 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.924117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.24.20.130:41825)
0620 20:17:29.924249 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:29.924253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:29.924265 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:17:29.924400 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:29.924402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:29.924452 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:29.924536 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:17:29.924810 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.925297 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.926103 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:29.926302 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:29.926374 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:29.926611 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:17:29.926979 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:17:29.927067 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:29.927122 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 20:17:29.927164 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:17:29.932756 11834 raft_consensus.cc:1261] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5 [term 1 FOLLOWER]: Refusing update from remote peer c8b3fd3b22ee45328853311ab94686e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:29.932757 11564 raft_consensus.cc:1261] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3 [term 1 FOLLOWER]: Refusing update from remote peer c8b3fd3b22ee45328853311ab94686e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:29.933461 11904 consensus_queue.cc:1035] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a7bc372b424ff68d7ac4f0d8e070e3" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 34555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:29.933655 11907 consensus_queue.cc:1035] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30537e7ce6194154a8c11f558c3befb5" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 36813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:29.941547 11910 mvcc.cc:205] Tried to move back new op lower bound from 7299017318117785600 to 7299017318030082048. Current Snapshot: MvccSnapshot[applied={T|T < 7299017318117785600}]
I20260620 20:17:29.941704 11911 mvcc.cc:205] Tried to move back new op lower bound from 7299017318117785600 to 7299017318030082048. Current Snapshot: MvccSnapshot[applied={T|T < 7299017318117785600}]
I20260620 20:17:29.951925 11420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:29.952077 11420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:29.954416 11420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edb4224e9beb43c2ab6c8dc0bff8aa00: Sending DeleteTablet for 3 replicas of tablet 1700c47495dd45a783dad70ed5aac6bf
I20260620 20:17:29.955163 11546 tablet_service.cc:1514] Processing DeleteTablet for tablet 1700c47495dd45a783dad70ed5aac6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:29 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:56896
I20260620 20:17:29.955161 11680 tablet_service.cc:1514] Processing DeleteTablet for tablet 1700c47495dd45a783dad70ed5aac6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:29 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:33754
I20260620 20:17:29.955216 11814 tablet_service.cc:1514] Processing DeleteTablet for tablet 1700c47495dd45a783dad70ed5aac6bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:29 UTC) from {username='kudu', principal='kudu/127.24.20.190@KRBTEST.COM'} at 127.0.0.1:50654
I20260620 20:17:29.955651 11921 tablet_replica.cc:330] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7: stopping tablet replica
I20260620 20:17:29.955973 11921 raft_consensus.cc:2229] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:17:29.956086 11922 tablet_replica.cc:330] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5: stopping tablet replica
I20260620 20:17:29.956382 11921 raft_consensus.cc:2258] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:29.956431 11922 raft_consensus.cc:2229] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:17:29.956791 11922 raft_consensus.cc:2258] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:29.956828 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 11483
I20260620 20:17:29.956885 11920 tablet_replica.cc:330] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3: stopping tablet replica
I20260620 20:17:29.957159 11920 raft_consensus.cc:2229] T 1700c47495dd45a783dad70ed5aac6bf P 34a7bc372b424ff68d7ac4f0d8e070e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:17:29.958334 11921 ts_tablet_manager.cc:1905] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:17:29.959249 11922 ts_tablet_manager.cc:1905] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:17:29.961833 11921 ts_tablet_manager.cc:1918] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:29.961921 11921 log.cc:1196] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/1700c47495dd45a783dad70ed5aac6bf
I20260620 20:17:29.961943 11922 ts_tablet_manager.cc:1918] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:29.962033 11922 log.cc:1196] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/1700c47495dd45a783dad70ed5aac6bf
I20260620 20:17:29.962273 11921 ts_tablet_manager.cc:1939] T 1700c47495dd45a783dad70ed5aac6bf P c8b3fd3b22ee45328853311ab94686e7: Deleting consensus metadata
I20260620 20:17:29.962409 11922 ts_tablet_manager.cc:1939] T 1700c47495dd45a783dad70ed5aac6bf P 30537e7ce6194154a8c11f558c3befb5: Deleting consensus metadata
I20260620 20:17:29.963629 11408 catalog_manager.cc:4841] TS 30537e7ce6194154a8c11f558c3befb5 (127.24.20.131:36813): tablet 1700c47495dd45a783dad70ed5aac6bf (table test-table [id=9ab6be18dd6f47589d33f492d80a7da7]) successfully deleted
I20260620 20:17:29.964496 11407 catalog_manager.cc:4841] TS c8b3fd3b22ee45328853311ab94686e7 (127.24.20.130:41825): tablet 1700c47495dd45a783dad70ed5aac6bf (table test-table [id=9ab6be18dd6f47589d33f492d80a7da7]) successfully deleted
W20260620 20:17:29.964933 11405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:34555 (error 108)
I20260620 20:17:29.965747 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 11617
I20260620 20:17:29.966084 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:29.965717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:34555 (local address 127.0.0.1:56906)
0620 20:17:29.965909 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:29.965984 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:34555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260620 20:17:29.966729 11405 catalog_manager.cc:4584] TS 34a7bc372b424ff68d7ac4f0d8e070e3 (127.24.20.129:34555): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1700c47495dd45a783dad70ed5aac6bf: Network error: Client connection negotiation failed: client connection to 127.24.20.129:34555: connect: Connection refused (error 111)
I20260620 20:17:29.971546 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 11751
I20260620 20:17:29.976887 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 11389
2026-06-20T20:17:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2266 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T20:17:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:29Z Disabled control of system clock
I20260620 20:17:30.006304 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:34893
--webserver_interface=127.24.20.190
--webserver_port=0
--builtin_ntp_servers=127.24.20.148:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:34893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:17:30.112426 11927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:30.112730 11927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:30.112769 11927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:30.116580 11927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:30.116686 11927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:30.116705 11927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:30.116722 11927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:30.116737 11927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:30.121342 11927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:34893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:34893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:30.122243 11927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:30.123471 11927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:30.131439 11935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:30.131456 11932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:30.131651 11933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:30.131800 11927 server_base.cc:1034] running on GCE node
I20260620 20:17:30.132355 11927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:30.133427 11927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:30.134583 11927 hybrid_clock.cc:648] HybridClock initialized: now 1781986650134557 us; error 40 us; skew 500 ppm
I20260620 20:17:30.136361 11927 webserver.cc:460] Webserver started at http://127.24.20.190:41071/ using document root <none> and password file <none>
I20260620 20:17:30.136968 11927 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:30.137028 11927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:30.137253 11927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:30.139601 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "94fb24a5b64d41f3a4b5a6057397d580"
format_stamp: "Formatted at 2026-06-20 20:17:30 on dist-test-slave-pflr"
server_key: "e96d90efe292dd9a10e3b5ab15c2d065"
server_key_iv: "4559b49e6dd3d212ba4bca5db67d8bd3"
server_key_version: "encryptionkey@0"
I20260620 20:17:30.140234 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "94fb24a5b64d41f3a4b5a6057397d580"
format_stamp: "Formatted at 2026-06-20 20:17:30 on dist-test-slave-pflr"
server_key: "e96d90efe292dd9a10e3b5ab15c2d065"
server_key_iv: "4559b49e6dd3d212ba4bca5db67d8bd3"
server_key_version: "encryptionkey@0"
I20260620 20:17:30.144009 11927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:17:30.146310 11941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:30.147390 11927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:17:30.147519 11927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "94fb24a5b64d41f3a4b5a6057397d580"
format_stamp: "Formatted at 2026-06-20 20:17:30 on dist-test-slave-pflr"
server_key: "e96d90efe292dd9a10e3b5ab15c2d065"
server_key_iv: "4559b49e6dd3d212ba4bca5db67d8bd3"
server_key_version: "encryptionkey@0"
I20260620 20:17:30.147634 11927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:30.157713 11927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:30.158313 11927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:30.158494 11927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:30.165041 11927 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:34893
I20260620 20:17:30.165046 11993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:34893 every 8 connection(s)
I20260620 20:17:30.165920 11927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:30.168933 11994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:30.171582 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 11927
I20260620 20:17:30.171728 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:30.172017 24658 external_mini_cluster.cc:1428] Setting key c347bac5c8b8f7b03ac99f813fe8fa4f
I20260620 20:17:30.174917 11994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580: Bootstrap starting.
I20260620 20:17:30.177393 11994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:30.178181 11994 log.cc:824] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:30.180372 11994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580: No bootstrap required, opened a new log
I20260620 20:17:30.181107 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.173351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.24.20.190:34893)
0620 20:17:30.173916 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:30.173930 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:30.173964 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:17:30.174548 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:30.174573 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:30.174940 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:30.175252 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:30.175584 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.176504 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.177720 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.178182 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.178336 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.179309 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:30.179314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:30.179327 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:30.179697 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:30.179752 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:30.179771 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:30.179845 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:30.180012 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 20:17:30.180299 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":116,"threads_started":1}
I20260620 20:17:30.184257 11994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fb24a5b64d41f3a4b5a6057397d580" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34893 } }
I20260620 20:17:30.184468 11994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:30.184544 11994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fb24a5b64d41f3a4b5a6057397d580, State: Initialized, Role: FOLLOWER
I20260620 20:17:30.184983 11994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fb24a5b64d41f3a4b5a6057397d580" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34893 } }
I20260620 20:17:30.185132 11994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:30.185197 11994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:30.185305 11994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:30.186298 11994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fb24a5b64d41f3a4b5a6057397d580" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34893 } }
I20260620 20:17:30.186645 11994 leader_election.cc:302] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94fb24a5b64d41f3a4b5a6057397d580; no voters:
I20260620 20:17:30.186940 11994 leader_election.cc:288] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:30.187088 11999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:30.187307 11999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [term 1 LEADER]: Becoming Leader. State: Replica: 94fb24a5b64d41f3a4b5a6057397d580, State: Running, Role: LEADER
I20260620 20:17:30.187623 11999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fb24a5b64d41f3a4b5a6057397d580" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34893 } }
I20260620 20:17:30.188151 11994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:30.189185 12000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94fb24a5b64d41f3a4b5a6057397d580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fb24a5b64d41f3a4b5a6057397d580" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34893 } } }
I20260620 20:17:30.189316 12000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:30.189697 12008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:30.189977 12001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94fb24a5b64d41f3a4b5a6057397d580. Latest consensus state: current_term: 1 leader_uuid: "94fb24a5b64d41f3a4b5a6057397d580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fb24a5b64d41f3a4b5a6057397d580" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34893 } } }
I20260620 20:17:30.190061 12001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:30.192726 12008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:30.198030 12008 catalog_manager.cc:1347] Generated new cluster ID: 736c9a2d4a7341a7851f2a9e549e90df
I20260620 20:17:30.198104 12008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:30.218075 12008 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:30.219033 12008 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:30.223358 12008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580: Generated new TSK 0
I20260620 20:17:30.224009 12008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:30.231794 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.129:0
--local_ip_for_outbound_sockets=127.24.20.129
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:34893
--builtin_ntp_servers=127.24.20.148:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:17:30.362074 12018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:30.362310 12018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:30.362370 12018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:30.366205 12018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:30.366251 12018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:30.366302 12018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.129
I20260620 20:17:30.370854 12018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.20.129
--webserver_port=0
--tserver_master_addrs=127.24.20.190:34893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.129
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:30.371757 12018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:30.373041 12018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:30.383103 12023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:30.383121 12026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:30.383359 12018 server_base.cc:1034] running on GCE node
W20260620 20:17:30.383369 12024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:30.383926 12018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:30.384569 12018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:30.385825 12018 hybrid_clock.cc:648] HybridClock initialized: now 1781986650385805 us; error 30 us; skew 500 ppm
I20260620 20:17:30.387737 12018 webserver.cc:460] Webserver started at http://127.24.20.129:36935/ using document root <none> and password file <none>
I20260620 20:17:30.388329 12018 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:30.388434 12018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:30.388756 12018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:30.390964 12018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/data/instance:
uuid: "470b34e51a074954a41302b95d90c8e2"
format_stamp: "Formatted at 2026-06-20 20:17:30 on dist-test-slave-pflr"
server_key: "3b841cb96a2883f8667251281bde860c"
server_key_iv: "2802b747266c9174b386d34281100947"
server_key_version: "encryptionkey@0"
I20260620 20:17:30.391564 12018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance:
uuid: "470b34e51a074954a41302b95d90c8e2"
format_stamp: "Formatted at 2026-06-20 20:17:30 on dist-test-slave-pflr"
server_key: "3b841cb96a2883f8667251281bde860c"
server_key_iv: "2802b747266c9174b386d34281100947"
server_key_version: "encryptionkey@0"
I20260620 20:17:30.395318 12018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:17:30.398105 12032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:30.399230 12018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:17:30.399381 12018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/wal
uuid: "470b34e51a074954a41302b95d90c8e2"
format_stamp: "Formatted at 2026-06-20 20:17:30 on dist-test-slave-pflr"
server_key: "3b841cb96a2883f8667251281bde860c"
server_key_iv: "2802b747266c9174b386d34281100947"
server_key_version: "encryptionkey@0"
I20260620 20:17:30.399494 12018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:30.426239 12018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:30.426882 12018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:30.427075 12018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:30.427629 12018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:30.428556 12018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:30.428646 12018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:30.428717 12018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:30.428764 12018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:30.437006 12018 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.129:40171
I20260620 20:17:30.437033 12145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.129:40171 every 8 connection(s)
I20260620 20:17:30.437839 12018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/data/info.pb
I20260620 20:17:30.438733 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 12018
I20260620 20:17:30.438866 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-0/wal/instance
I20260620 20:17:30.439209 24658 external_mini_cluster.cc:1428] Setting key 11ae36934002a9d24c587b0231f4ac26
I20260620 20:17:30.441396 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.130:0
--local_ip_for_outbound_sockets=127.24.20.130
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:34893
--builtin_ntp_servers=127.24.20.148:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:17:30.447798 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.439656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.129:58353 (local address 127.24.20.190:34893)
0620 20:17:30.439814 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:30.439819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:30.441132 (+ 1313us) server_negotiation.cc:407] Connection header received
0620 20:17:30.443145 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:30.443148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:30.443203 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:30.443304 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:30.444148 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.444730 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.445902 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.446132 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.446244 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.447320 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:30.447323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:30.447326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:30.447338 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:30.447363 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:30.447371 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:30.447444 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:30.447607 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 20:17:30.447663 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:17:30.448678 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.440147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:34893 (local address 127.24.20.129:58353)
0620 20:17:30.440978 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:30.441028 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:30.442911 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:30.443474 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:30.443481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:30.443565 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:30.444010 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:30.444027 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.444847 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.444850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:30.445779 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:30.445788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.446279 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.446283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:30.446450 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.447063 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:30.447065 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:30.447103 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:30.447161 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:30.447458 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:30.447467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:30.447472 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:17:30.447606 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 20:17:30.447856 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":140,"threads_started":1}
I20260620 20:17:30.449854 12146 heartbeater.cc:347] Connected to a master server at 127.24.20.190:34893
I20260620 20:17:30.450085 12146 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:30.450683 12146 heartbeater.cc:511] Master 127.24.20.190:34893 requested a full tablet report, sending...
I20260620 20:17:30.452188 11958 ts_manager.cc:194] Registered new tserver with Master: 470b34e51a074954a41302b95d90c8e2 (127.24.20.129:40171)
I20260620 20:17:30.453310 11958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.20.129:58353
W20260620 20:17:30.551882 12149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:30.552160 12149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:30.552227 12149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:30.556176 12149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:30.556226 12149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:30.556277 12149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.130
I20260620 20:17:30.560882 12149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.20.130
--webserver_port=0
--tserver_master_addrs=127.24.20.190:34893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.130
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:30.561801 12149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:30.563135 12149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:30.572132 12154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:30.572132 12157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:30.572099 12155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:30.572201 12149 server_base.cc:1034] running on GCE node
I20260620 20:17:30.572825 12149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:30.573400 12149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:30.574554 12149 hybrid_clock.cc:648] HybridClock initialized: now 1781986650574542 us; error 27 us; skew 500 ppm
I20260620 20:17:30.576233 12149 webserver.cc:460] Webserver started at http://127.24.20.130:37005/ using document root <none> and password file <none>
I20260620 20:17:30.576849 12149 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:30.576931 12149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:30.577158 12149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:30.579265 12149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/data/instance:
uuid: "2a3d33f1b191405db44337cd6c24c85b"
format_stamp: "Formatted at 2026-06-20 20:17:30 on dist-test-slave-pflr"
server_key: "2fed17a53304472c0b8eb11eebc5f524"
server_key_iv: "6cf326efdd483d4336b0b17eafaae49b"
server_key_version: "encryptionkey@0"
I20260620 20:17:30.579818 12149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance:
uuid: "2a3d33f1b191405db44337cd6c24c85b"
format_stamp: "Formatted at 2026-06-20 20:17:30 on dist-test-slave-pflr"
server_key: "2fed17a53304472c0b8eb11eebc5f524"
server_key_iv: "6cf326efdd483d4336b0b17eafaae49b"
server_key_version: "encryptionkey@0"
I20260620 20:17:30.583244 12149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:17:30.585485 12163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:30.586560 12149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:17:30.586684 12149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/wal
uuid: "2a3d33f1b191405db44337cd6c24c85b"
format_stamp: "Formatted at 2026-06-20 20:17:30 on dist-test-slave-pflr"
server_key: "2fed17a53304472c0b8eb11eebc5f524"
server_key_iv: "6cf326efdd483d4336b0b17eafaae49b"
server_key_version: "encryptionkey@0"
I20260620 20:17:30.586781 12149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:30.603232 12149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:30.603816 12149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:30.603992 12149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:30.604513 12149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:30.605391 12149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:30.605437 12149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:30.605500 12149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:30.605540 12149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:30.614550 12149 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.130:42145
I20260620 20:17:30.614583 12276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.130:42145 every 8 connection(s)
I20260620 20:17:30.615433 12149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/data/info.pb
I20260620 20:17:30.617503 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 12149
I20260620 20:17:30.617600 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/wal/instance
I20260620 20:17:30.618023 24658 external_mini_cluster.cc:1428] Setting key 05c73d8f192e6d0621a49b34c1efdf0e
I20260620 20:17:30.620234 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.20.131:0
--local_ip_for_outbound_sockets=127.24.20.131
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:34893
--builtin_ntp_servers=127.24.20.148:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:17:30.625221 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.617136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:40103 (local address 127.24.20.190:34893)
0620 20:17:30.617267 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:30.617270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:30.618441 (+ 1171us) server_negotiation.cc:407] Connection header received
0620 20:17:30.620054 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:30.620057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:30.620099 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:30.620165 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:30.621289 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.621811 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.623290 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.623492 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.623613 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.624750 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:30.624754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:30.624756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:30.624767 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:30.624784 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:30.624791 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:30.624866 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:30.625027 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 20:17:30.625086 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:17:30.626003 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.617460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:34893 (local address 127.24.20.130:40103)
0620 20:17:30.618273 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:30.618322 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:30.619844 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:30.620336 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:30.620346 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:30.620454 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:30.621150 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:30.621171 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.621935 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.621940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:30.623191 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:30.623201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.623650 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.623654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:30.623825 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.624445 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:30.624447 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:30.624484 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:30.624542 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:30.624881 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:30.624889 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:30.624894 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:17:30.625017 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 20:17:30.625286 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":170,"threads_started":1}
I20260620 20:17:30.627199 12277 heartbeater.cc:347] Connected to a master server at 127.24.20.190:34893
I20260620 20:17:30.627414 12277 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:30.627877 12277 heartbeater.cc:511] Master 127.24.20.190:34893 requested a full tablet report, sending...
I20260620 20:17:30.628917 11958 ts_manager.cc:194] Registered new tserver with Master: 2a3d33f1b191405db44337cd6c24c85b (127.24.20.130:42145)
I20260620 20:17:30.629468 11958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.20.130:40103
W20260620 20:17:30.729887 12280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:30.730139 12280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:30.730203 12280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:30.734097 12280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:30.734163 12280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:30.734241 12280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.20.131
I20260620 20:17:30.738895 12280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.20.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.20.131
--webserver_port=0
--tserver_master_addrs=127.24.20.190:34893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.20.131
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:30.739904 12280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:30.741276 12280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:30.750389 12288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:30.750417 12286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:30.750511 12280 server_base.cc:1034] running on GCE node
W20260620 20:17:30.750432 12285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:30.751297 12280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:30.751919 12280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:30.753069 12280 hybrid_clock.cc:648] HybridClock initialized: now 1781986650753056 us; error 27 us; skew 500 ppm
I20260620 20:17:30.754781 12280 webserver.cc:460] Webserver started at http://127.24.20.131:34877/ using document root <none> and password file <none>
I20260620 20:17:30.755342 12280 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:30.755401 12280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:30.755656 12280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:30.757751 12280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/data/instance:
uuid: "bf674a77c5b54056b52b94ed5b3219f4"
format_stamp: "Formatted at 2026-06-20 20:17:30 on dist-test-slave-pflr"
server_key: "f0007e335f3fda12d27a544cd7c39fbe"
server_key_iv: "d77cafcd8aef89327d8df21d24cafa63"
server_key_version: "encryptionkey@0"
I20260620 20:17:30.758329 12280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance:
uuid: "bf674a77c5b54056b52b94ed5b3219f4"
format_stamp: "Formatted at 2026-06-20 20:17:30 on dist-test-slave-pflr"
server_key: "f0007e335f3fda12d27a544cd7c39fbe"
server_key_iv: "d77cafcd8aef89327d8df21d24cafa63"
server_key_version: "encryptionkey@0"
I20260620 20:17:30.761915 12280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:30.764205 12294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:30.765342 12280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:30.765471 12280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/wal
uuid: "bf674a77c5b54056b52b94ed5b3219f4"
format_stamp: "Formatted at 2026-06-20 20:17:30 on dist-test-slave-pflr"
server_key: "f0007e335f3fda12d27a544cd7c39fbe"
server_key_iv: "d77cafcd8aef89327d8df21d24cafa63"
server_key_version: "encryptionkey@0"
I20260620 20:17:30.765609 12280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:30.787606 12280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:30.788306 12280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:30.788501 12280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:30.789191 12280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:17:30.790210 12280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:17:30.790278 12280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:30.790350 12280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:17:30.790405 12280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:30.799321 12280 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.131:35377
I20260620 20:17:30.799360 12407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.131:35377 every 8 connection(s)
I20260620 20:17:30.800251 12280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/data/info.pb
I20260620 20:17:30.807276 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 12280
I20260620 20:17:30.807367 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/wal/instance
I20260620 20:17:30.807695 24658 external_mini_cluster.cc:1428] Setting key da2a54197515f038f8507e66fde9b594
I20260620 20:17:30.809741 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.802113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.131:42109 (local address 127.24.20.190:34893)
0620 20:17:30.802284 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:30.802289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:30.803451 (+ 1162us) server_negotiation.cc:407] Connection header received
0620 20:17:30.804911 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:30.804914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:30.804962 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:30.805038 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:30.805885 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.806396 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.807518 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.807854 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.807955 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.809249 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:30.809252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:30.809255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:30.809274 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:30.809293 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:30.809301 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:30.809375 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:30.809537 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 20:17:30.809604 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:17:30.810555 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.802639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:34893 (local address 127.24.20.131:42109)
0620 20:17:30.803301 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:30.803342 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:30.804725 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:30.805210 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:30.805216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:30.805298 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:30.805767 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:30.805779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.806501 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.806503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:30.807413 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:30.807425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.808029 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.808034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:30.808239 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.808954 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:17:30.808956 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:30.809007 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:17:30.809106 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:30.809389 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:30.809398 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:30.809403 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:17:30.809537 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 20:17:30.809800 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
I20260620 20:17:30.811573 12408 heartbeater.cc:347] Connected to a master server at 127.24.20.190:34893
I20260620 20:17:30.811797 12408 heartbeater.cc:464] Registering TS with master...
I20260620 20:17:30.812274 12408 heartbeater.cc:511] Master 127.24.20.190:34893 requested a full tablet report, sending...
I20260620 20:17:30.813270 11958 ts_manager.cc:194] Registered new tserver with Master: bf674a77c5b54056b52b94ed5b3219f4 (127.24.20.131:35377)
I20260620 20:17:30.813817 11958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.20.131:42109
I20260620 20:17:30.821441 24658 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:17:30.826681 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.823303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.24.20.190:34893)
0620 20:17:30.823434 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:30.823438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:30.823624 (+ 186us) server_negotiation.cc:407] Connection header received
0620 20:17:30.823708 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:30.823710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:30.823746 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:30.823808 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:30.824020 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.824503 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.825331 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.825531 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.825614 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.826293 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:30.826296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:30.826298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:30.826309 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:30.826326 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:30.826333 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:30.826389 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:30.826463 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 20:17:30.826505 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:17:30.829288 11958 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:17:30.832304 11958 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:17:30.845973 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.840263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.24.20.129:40171)
0620 20:17:30.840614 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:30.840631 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:30.841365 (+ 734us) server_negotiation.cc:407] Connection header received
0620 20:17:30.841452 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:30.841460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:30.841707 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:30.841872 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:30.842167 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.842942 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.844446 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.845278 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.845387 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.845581 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:30.845696 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 20:17:30.845799 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":126,"threads_started":1}
I20260620 20:17:30.846000 12419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.840097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:40171 (local address 127.0.0.1:39710)
0620 20:17:30.840758 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:30.840808 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:30.840943 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:30.841860 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:30.841866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:30.841887 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:30.842025 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:30.842031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.843069 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.843073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:30.844309 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:30.844320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.845404 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.845408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:30.845514 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.845550 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 20:17:30.845666 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 20:17:30.845783 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":33,"thread_start_us":63,"threads_started":1}
I20260620 20:17:30.846210 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.840290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:35377 (local address 127.0.0.1:49116)
0620 20:17:30.841091 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:30.841106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:30.841195 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:30.841934 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:30.841940 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:30.841963 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:30.842061 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:30.842069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.843560 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.843567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:30.844784 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:30.844794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.845791 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.845794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:30.845884 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.845906 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:17:30.845952 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 20:17:30.846043 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":33,"thread_start_us":103,"threads_started":1}
I20260620 20:17:30.846338 12424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.841218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.24.20.130:42145)
0620 20:17:30.841629 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:30.841640 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:30.841659 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:17:30.841722 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:30.841725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:30.841880 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:30.842015 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:30.842222 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.843409 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.844850 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.845392 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.845495 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.845689 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:30.846068 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 20:17:30.846170 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":110,"threads_started":1}
I20260620 20:17:30.846437 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.840813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:42145 (local address 127.0.0.1:50572)
0620 20:17:30.841161 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:30.841175 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:30.841257 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:30.842023 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:30.842026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:30.842041 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:30.842131 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:30.842136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.843531 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.843535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:30.844734 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:30.844742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.845745 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.845749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:30.845822 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.845859 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:17:30.845928 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 20:17:30.845994 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":56,"thread_start_us":87,"threads_started":1}
I20260620 20:17:30.846927 12423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:30.840831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.24.20.131:35377)
0620 20:17:30.841457 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:30.841468 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:30.841492 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:17:30.841580 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:30.841585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:30.841777 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:30.842202 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:30.842335 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.843429 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.844917 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:30.845659 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:30.846450 (+ 791us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:30.846629 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:30.846682 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 20:17:30.846771 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":117,"threads_started":1}
I20260620 20:17:30.849143 12080 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea8497ff46c46d5a23764ac56526a96 (DEFAULT_TABLE table=test-table [id=dbe18442c4e740f9b6f7c0f15db1de1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:30.849236 12211 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea8497ff46c46d5a23764ac56526a96 (DEFAULT_TABLE table=test-table [id=dbe18442c4e740f9b6f7c0f15db1de1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:30.849969 12080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea8497ff46c46d5a23764ac56526a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:30.850188 12211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea8497ff46c46d5a23764ac56526a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:30.850301 12342 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea8497ff46c46d5a23764ac56526a96 (DEFAULT_TABLE table=test-table [id=dbe18442c4e740f9b6f7c0f15db1de1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:17:30.851187 12342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea8497ff46c46d5a23764ac56526a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:30.855795 12425 tablet_bootstrap.cc:492] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2: Bootstrap starting.
I20260620 20:17:30.857110 12426 tablet_bootstrap.cc:492] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b: Bootstrap starting.
I20260620 20:17:30.858150 12425 tablet_bootstrap.cc:654] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:30.859030 12427 tablet_bootstrap.cc:492] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4: Bootstrap starting.
I20260620 20:17:30.859139 12426 tablet_bootstrap.cc:654] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:30.859139 12425 log.cc:824] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:30.859874 12426 log.cc:824] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:30.861011 12427 tablet_bootstrap.cc:654] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:30.861644 12427 log.cc:824] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:30.861732 12425 tablet_bootstrap.cc:492] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2: No bootstrap required, opened a new log
I20260620 20:17:30.861841 12426 tablet_bootstrap.cc:492] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b: No bootstrap required, opened a new log
I20260620 20:17:30.861915 12425 ts_tablet_manager.cc:1397] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 20:17:30.861985 12426 ts_tablet_manager.cc:1397] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:17:30.863300 12427 tablet_bootstrap.cc:492] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4: No bootstrap required, opened a new log
I20260620 20:17:30.863479 12427 ts_tablet_manager.cc:1397] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:17:30.866149 12425 raft_consensus.cc:348] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b34e51a074954a41302b95d90c8e2" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 40171 } } peers { permanent_uuid: "bf674a77c5b54056b52b94ed5b3219f4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35377 } } peers { permanent_uuid: "2a3d33f1b191405db44337cd6c24c85b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42145 } }
I20260620 20:17:30.866117 12426 raft_consensus.cc:348] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b34e51a074954a41302b95d90c8e2" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 40171 } } peers { permanent_uuid: "bf674a77c5b54056b52b94ed5b3219f4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35377 } } peers { permanent_uuid: "2a3d33f1b191405db44337cd6c24c85b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42145 } }
I20260620 20:17:30.866379 12425 raft_consensus.cc:374] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:30.866379 12426 raft_consensus.cc:374] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:30.866433 12425 raft_consensus.cc:729] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 470b34e51a074954a41302b95d90c8e2, State: Initialized, Role: FOLLOWER
I20260620 20:17:30.866433 12426 raft_consensus.cc:729] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3d33f1b191405db44337cd6c24c85b, State: Initialized, Role: FOLLOWER
I20260620 20:17:30.866549 12427 raft_consensus.cc:348] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b34e51a074954a41302b95d90c8e2" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 40171 } } peers { permanent_uuid: "bf674a77c5b54056b52b94ed5b3219f4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35377 } } peers { permanent_uuid: "2a3d33f1b191405db44337cd6c24c85b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42145 } }
I20260620 20:17:30.866829 12427 raft_consensus.cc:374] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:30.866894 12427 raft_consensus.cc:729] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf674a77c5b54056b52b94ed5b3219f4, State: Initialized, Role: FOLLOWER
I20260620 20:17:30.866868 12426 consensus_queue.cc:260] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b34e51a074954a41302b95d90c8e2" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 40171 } } peers { permanent_uuid: "bf674a77c5b54056b52b94ed5b3219f4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35377 } } peers { permanent_uuid: "2a3d33f1b191405db44337cd6c24c85b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42145 } }
I20260620 20:17:30.866868 12425 consensus_queue.cc:260] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b34e51a074954a41302b95d90c8e2" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 40171 } } peers { permanent_uuid: "bf674a77c5b54056b52b94ed5b3219f4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35377 } } peers { permanent_uuid: "2a3d33f1b191405db44337cd6c24c85b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42145 } }
I20260620 20:17:30.867311 12427 consensus_queue.cc:260] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b34e51a074954a41302b95d90c8e2" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 40171 } } peers { permanent_uuid: "bf674a77c5b54056b52b94ed5b3219f4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35377 } } peers { permanent_uuid: "2a3d33f1b191405db44337cd6c24c85b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42145 } }
I20260620 20:17:30.867498 12146 heartbeater.cc:503] Master 127.24.20.190:34893 was elected leader, sending a full tablet report...
I20260620 20:17:30.867815 12426 ts_tablet_manager.cc:1428] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 20:17:30.868127 12277 heartbeater.cc:503] Master 127.24.20.190:34893 was elected leader, sending a full tablet report...
I20260620 20:17:30.868227 12427 ts_tablet_manager.cc:1428] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:17:30.868527 12408 heartbeater.cc:503] Master 127.24.20.190:34893 was elected leader, sending a full tablet report...
I20260620 20:17:30.871148 12425 ts_tablet_manager.cc:1428] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
W20260620 20:17:30.938930 12147 tablet.cc:2369] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:17:31.051323 12409 tablet.cc:2369] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:17:31.065649 12431 raft_consensus.cc:482] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:17:31.065881 12431 raft_consensus.cc:504] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b34e51a074954a41302b95d90c8e2" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 40171 } } peers { permanent_uuid: "bf674a77c5b54056b52b94ed5b3219f4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35377 } } peers { permanent_uuid: "2a3d33f1b191405db44337cd6c24c85b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42145 } }
I20260620 20:17:31.067080 12431 leader_election.cc:288] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 470b34e51a074954a41302b95d90c8e2 (127.24.20.129:40171), bf674a77c5b54056b52b94ed5b3219f4 (127.24.20.131:35377)
I20260620 20:17:31.071308 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:31.067401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.131:35377 (local address 127.24.20.130:58455)
0620 20:17:31.067515 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:31.067533 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:31.067632 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:31.067932 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:31.067935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:31.067950 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:31.068106 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:31.068115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:31.069082 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:31.069086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:31.070033 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:31.070043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:31.070878 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:31.070882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:31.070978 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:31.071004 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 20:17:31.071100 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 20:17:31.071168 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 20:17:31.071349 12423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:31.067512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:58455 (local address 127.24.20.131:35377)
0620 20:17:31.067640 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:31.067645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:31.067663 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:17:31.067756 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:31.067759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:31.067810 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:31.067911 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:31.068231 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:31.068971 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:31.070160 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:31.070762 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:31.070863 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:31.070905 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:31.071130 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 20:17:31.071224 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:17:31.071766 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:31.067504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.20.130:52975 (local address 127.24.20.129:40171)
0620 20:17:31.067639 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:31.067644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:31.068051 (+ 407us) server_negotiation.cc:407] Connection header received
0620 20:17:31.068144 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:31.068150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:31.068214 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:31.068284 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:17:31.068561 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:31.069213 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:31.070614 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:31.071142 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:31.071250 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:31.071290 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:31.071537 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 20:17:31.071652 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:17:31.072103 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:31.067401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:40171 (local address 127.24.20.130:52975)
0620 20:17:31.067940 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:31.067954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:31.068062 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:31.068319 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:31.068322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:31.068339 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:17:31.068453 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:31.068459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:31.069476 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:31.069479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:31.070477 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:31.070487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:31.071279 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:31.071283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:31.071380 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:31.071404 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:17:31.071922 (+ 518us) client_negotiation.cc:240] Negotiation successful
0620 20:17:31.071998 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":144,"thread_start_us":124,"threads_started":1}
I20260620 20:17:31.072098 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea8497ff46c46d5a23764ac56526a96" candidate_uuid: "2a3d33f1b191405db44337cd6c24c85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf674a77c5b54056b52b94ed5b3219f4" is_pre_election: true
I20260620 20:17:31.072427 12362 raft_consensus.cc:2454] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a3d33f1b191405db44337cd6c24c85b in term 0.
I20260620 20:17:31.072741 12100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea8497ff46c46d5a23764ac56526a96" candidate_uuid: "2a3d33f1b191405db44337cd6c24c85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "470b34e51a074954a41302b95d90c8e2" is_pre_election: true
I20260620 20:17:31.072996 12164 leader_election.cc:302] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a3d33f1b191405db44337cd6c24c85b, bf674a77c5b54056b52b94ed5b3219f4; no voters:
I20260620 20:17:31.073047 12100 raft_consensus.cc:2454] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a3d33f1b191405db44337cd6c24c85b in term 0.
I20260620 20:17:31.073287 12431 raft_consensus.cc:2781] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:17:31.073391 12431 raft_consensus.cc:482] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:17:31.073451 12431 raft_consensus.cc:3037] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:31.074460 12431 raft_consensus.cc:504] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b34e51a074954a41302b95d90c8e2" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 40171 } } peers { permanent_uuid: "bf674a77c5b54056b52b94ed5b3219f4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35377 } } peers { permanent_uuid: "2a3d33f1b191405db44337cd6c24c85b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42145 } }
I20260620 20:17:31.074795 12431 leader_election.cc:288] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [CANDIDATE]: Term 1 election: Requested vote from peers 470b34e51a074954a41302b95d90c8e2 (127.24.20.129:40171), bf674a77c5b54056b52b94ed5b3219f4 (127.24.20.131:35377)
I20260620 20:17:31.075122 12362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea8497ff46c46d5a23764ac56526a96" candidate_uuid: "2a3d33f1b191405db44337cd6c24c85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf674a77c5b54056b52b94ed5b3219f4"
I20260620 20:17:31.075143 12100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea8497ff46c46d5a23764ac56526a96" candidate_uuid: "2a3d33f1b191405db44337cd6c24c85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "470b34e51a074954a41302b95d90c8e2"
I20260620 20:17:31.075248 12100 raft_consensus.cc:3037] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:31.075248 12362 raft_consensus.cc:3037] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:31.076452 12362 raft_consensus.cc:2454] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a3d33f1b191405db44337cd6c24c85b in term 1.
I20260620 20:17:31.076452 12100 raft_consensus.cc:2454] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a3d33f1b191405db44337cd6c24c85b in term 1.
I20260620 20:17:31.076762 12166 leader_election.cc:302] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a3d33f1b191405db44337cd6c24c85b, 470b34e51a074954a41302b95d90c8e2; no voters:
I20260620 20:17:31.076961 12431 raft_consensus.cc:2781] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:31.077171 12431 raft_consensus.cc:686] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [term 1 LEADER]: Becoming Leader. State: Replica: 2a3d33f1b191405db44337cd6c24c85b, State: Running, Role: LEADER
I20260620 20:17:31.077481 12431 consensus_queue.cc:237] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b34e51a074954a41302b95d90c8e2" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 40171 } } peers { permanent_uuid: "bf674a77c5b54056b52b94ed5b3219f4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35377 } } peers { permanent_uuid: "2a3d33f1b191405db44337cd6c24c85b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42145 } }
I20260620 20:17:31.081218 11956 catalog_manager.cc:5495] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a3d33f1b191405db44337cd6c24c85b (127.24.20.130). New cstate: current_term: 1 leader_uuid: "2a3d33f1b191405db44337cd6c24c85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "470b34e51a074954a41302b95d90c8e2" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf674a77c5b54056b52b94ed5b3219f4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a3d33f1b191405db44337cd6c24c85b" member_type: VOTER last_known_addr { host: "127.24.20.130" port: 42145 } health_report { overall_health: HEALTHY } } }
W20260620 20:17:31.116506 12278 tablet.cc:2369] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:17:31.144063 12424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:31.140690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50578 (local address 127.24.20.130:42145)
0620 20:17:31.140817 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:31.140821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:31.140835 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:17:31.140938 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:31.140940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:31.140996 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:31.141082 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:17:31.141303 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:31.141792 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:31.142634 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:31.142828 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:31.142901 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:31.143162 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:17:31.143563 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:17:31.143661 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:31.143830 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 20:17:31.143872 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:17:31.149627 12362 raft_consensus.cc:1261] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4 [term 1 FOLLOWER]: Refusing update from remote peer 2a3d33f1b191405db44337cd6c24c85b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:31.149628 12100 raft_consensus.cc:1261] T 4ea8497ff46c46d5a23764ac56526a96 P 470b34e51a074954a41302b95d90c8e2 [term 1 FOLLOWER]: Refusing update from remote peer 2a3d33f1b191405db44337cd6c24c85b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:31.150389 12435 consensus_queue.cc:1035] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [LEADER]: Connected to new peer: Peer: permanent_uuid: "470b34e51a074954a41302b95d90c8e2" member_type: VOTER last_known_addr { host: "127.24.20.129" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:31.150645 12431 consensus_queue.cc:1035] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf674a77c5b54056b52b94ed5b3219f4" member_type: VOTER last_known_addr { host: "127.24.20.131" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:31.157570 12439 mvcc.cc:205] Tried to move back new op lower bound from 7299017323101609984 to 7299017322815799296. Current Snapshot: MvccSnapshot[applied={T|T < 7299017323101609984}]
I20260620 20:17:31.167496 11958 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:31.167666 11958 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:17:31.170151 11958 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94fb24a5b64d41f3a4b5a6057397d580: Sending DeleteTablet for 3 replicas of tablet 4ea8497ff46c46d5a23764ac56526a96
I20260620 20:17:31.170918 12080 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea8497ff46c46d5a23764ac56526a96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:31 UTC) from {username='slave'} at 127.0.0.1:39710
I20260620 20:17:31.170944 12342 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea8497ff46c46d5a23764ac56526a96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:31 UTC) from {username='slave'} at 127.0.0.1:49116
I20260620 20:17:31.171538 12211 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea8497ff46c46d5a23764ac56526a96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:17:31 UTC) from {username='slave'} at 127.0.0.1:50572
I20260620 20:17:31.171788 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 12018
I20260620 20:17:31.176801 12447 tablet_replica.cc:330] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4: stopping tablet replica
I20260620 20:17:31.177110 12447 raft_consensus.cc:2229] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:17:31.177418 12447 raft_consensus.cc:2258] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:31.178707 12447 ts_tablet_manager.cc:1905] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:17:31.180764 12449 tablet_replica.cc:330] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b: stopping tablet replica
I20260620 20:17:31.180929 12447 ts_tablet_manager.cc:1918] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:31.181030 12447 log.cc:1196] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-2/wal/wals/4ea8497ff46c46d5a23764ac56526a96
I20260620 20:17:31.181052 12449 raft_consensus.cc:2229] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:17:31.181341 12447 ts_tablet_manager.cc:1939] T 4ea8497ff46c46d5a23764ac56526a96 P bf674a77c5b54056b52b94ed5b3219f4: Deleting consensus metadata
I20260620 20:17:31.181443 12449 raft_consensus.cc:2258] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:17:31.182631 11942 catalog_manager.cc:4841] TS bf674a77c5b54056b52b94ed5b3219f4 (127.24.20.131:35377): tablet 4ea8497ff46c46d5a23764ac56526a96 (table test-table [id=dbe18442c4e740f9b6f7c0f15db1de1b]) successfully deleted
I20260620 20:17:31.183575 12449 ts_tablet_manager.cc:1905] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 20:17:31.185727 11944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.20.129:40171 (error 108)
I20260620 20:17:31.186185 12449 ts_tablet_manager.cc:1918] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:17:31.186288 12449 log.cc:1196] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b: Deleting WAL directory at /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986405628821-24658-0/minicluster-data/ts-1/wal/wals/4ea8497ff46c46d5a23764ac56526a96
I20260620 20:17:31.186604 12449 ts_tablet_manager.cc:1939] T 4ea8497ff46c46d5a23764ac56526a96 P 2a3d33f1b191405db44337cd6c24c85b: Deleting consensus metadata
I20260620 20:17:31.186830 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:31.186534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.129:40171 (local address 127.0.0.1:39716)
0620 20:17:31.186661 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:31.186737 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.129:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 20:17:31.186869 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 12149
W20260620 20:17:31.188261 11944 catalog_manager.cc:4584] TS 470b34e51a074954a41302b95d90c8e2 (127.24.20.129:40171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ea8497ff46c46d5a23764ac56526a96: Network error: Client connection negotiation failed: client connection to 127.24.20.129:40171: connect: Connection refused (error 111)
I20260620 20:17:31.191808 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 12280
I20260620 20:17:31.192631 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:31.191982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.130:42145 (local address 127.0.0.1:50592)
0620 20:17:31.192446 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:31.192509 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.130:42145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356}
I20260620 20:17:31.197438 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 11927
2026-06-20T20:17:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1219 ms)
[----------] 6 tests from EncryptionPolicyTest (10293 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:31 dist-test-slave-pflr krb5kdc[12453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:31 dist-test-slave-pflr krb5kdc[12453](info): set up 2 sockets
Jun 20 20:17:31 dist-test-slave-pflr krb5kdc[12453](info): commencing operation
krb5kdc: starting...
W20260620 20:17:32.232998 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:32 dist-test-slave-pflr krb5kdc[12453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:17:32.351764 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:32.457695 12466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:32.457963 12466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:32.458034 12466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:32.461958 12466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:32.462035 12466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:32.462067 12466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:32.462105 12466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:32.462142 12466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:32.467164 12466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:32.468150 12466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:32.469185 12466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:17:32.477344 12466 server_base.cc:1034] running on GCE node
W20260620 20:17:32.477512 12471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:32.477521 12474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:32.477540 12472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:32.478245 12466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:32.479290 12466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:32.480484 12466 hybrid_clock.cc:648] HybridClock initialized: now 1781986652480452 us; error 45 us; skew 500 ppm
Jun 20 20:17:32 dist-test-slave-pflr krb5kdc[12453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:32.483805 12466 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:17:32.484944 12466 webserver.cc:460] Webserver started at http://127.0.0.1:43933/ using document root <none> and password file <none>
I20260620 20:17:32.485476 12466 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:32.485570 12466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:32.485770 12466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:32.487769 12466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "58182c495175466e8d91a66e67d89358"
format_stamp: "Formatted at 2026-06-20 20:17:32 on dist-test-slave-pflr"
server_key: "1b9ca0e8761464751e5b98004aaf2278"
server_key_iv: "9589192c3a0202112ee1e259215b61b7"
server_key_version: "encryptionkey@0"
I20260620 20:17:32.488313 12466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "58182c495175466e8d91a66e67d89358"
format_stamp: "Formatted at 2026-06-20 20:17:32 on dist-test-slave-pflr"
server_key: "1b9ca0e8761464751e5b98004aaf2278"
server_key_iv: "9589192c3a0202112ee1e259215b61b7"
server_key_version: "encryptionkey@0"
I20260620 20:17:32.491781 12466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:17:32.494098 12481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:32.495237 12466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:17:32.495370 12466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "58182c495175466e8d91a66e67d89358"
format_stamp: "Formatted at 2026-06-20 20:17:32 on dist-test-slave-pflr"
server_key: "1b9ca0e8761464751e5b98004aaf2278"
server_key_iv: "9589192c3a0202112ee1e259215b61b7"
server_key_version: "encryptionkey@0"
I20260620 20:17:32.495476 12466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:32.519526 12466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:32.522858 12466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:32.523069 12466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:32.530166 12466 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45239
I20260620 20:17:32.530194 12533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45239 every 8 connection(s)
I20260620 20:17:32.531112 12466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:32.533779 12534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:32.537381 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 12466
I20260620 20:17:32.537528 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:32.537807 24658 external_mini_cluster.cc:1428] Setting key 31b68ac25c3e4e5f3471b22a60850852
I20260620 20:17:32.539503 12534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358: Bootstrap starting.
I20260620 20:17:32.541607 12534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:32.542387 12534 log.cc:824] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:32.544133 12534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358: No bootstrap required, opened a new log
Jun 20 20:17:32 dist-test-slave-pflr krb5kdc[12453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:17:32.547887 12534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58182c495175466e8d91a66e67d89358" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } }
I20260620 20:17:32.548115 12534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:32.548198 12534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58182c495175466e8d91a66e67d89358, State: Initialized, Role: FOLLOWER
I20260620 20:17:32.548702 12534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58182c495175466e8d91a66e67d89358" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } }
I20260620 20:17:32.548858 12534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:32.548960 12534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:32.549064 12534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:32.549917 12534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58182c495175466e8d91a66e67d89358" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } }
I20260620 20:17:32.550271 12534 leader_election.cc:302] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58182c495175466e8d91a66e67d89358; no voters:
I20260620 20:17:32.550623 12534 leader_election.cc:288] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:32.550705 12539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:32.550951 12539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [term 1 LEADER]: Becoming Leader. State: Replica: 58182c495175466e8d91a66e67d89358, State: Running, Role: LEADER
I20260620 20:17:32.551266 12539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58182c495175466e8d91a66e67d89358" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } }
I20260620 20:17:32.551909 12534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:32.553205 12540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58182c495175466e8d91a66e67d89358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58182c495175466e8d91a66e67d89358" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } } }
I20260620 20:17:32.553333 12540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:32.553620 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:32.539251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60799 (local address 127.0.0.1:45239)
0620 20:17:32.539739 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:32.539757 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:32.539793 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:17:32.540406 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:32.540423 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:32.540822 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:32.541187 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:32.542080 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:32.543172 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:32.544200 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:32.544828 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:32.545009 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:32.547809 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:32.547815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:32.547828 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:32.547855 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:32.549923 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:32.550574 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:32.550581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:32.550588 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:32.550732 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:32.551165 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:32.551168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:32.551169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:32.551632 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:32.551834 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:32.552224 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 20:17:32.552450 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":133,"threads_started":1}
I20260620 20:17:32.553632 12541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58182c495175466e8d91a66e67d89358. Latest consensus state: current_term: 1 leader_uuid: "58182c495175466e8d91a66e67d89358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58182c495175466e8d91a66e67d89358" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45239 } } }
I20260620 20:17:32.553984 12541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:32.553830 12550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:32.557324 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:32.557332 12550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:32.563189 12550 catalog_manager.cc:1347] Generated new cluster ID: 0bd2eb25dbd54b6b8f9d434bf5d92b6c
I20260620 20:17:32.563324 12550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:32.568745 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:32.559589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48655 (local address 127.0.0.1:45239)
0620 20:17:32.559711 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:32.559715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:32.560052 (+ 337us) server_negotiation.cc:407] Connection header received
0620 20:17:32.560154 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:32.560157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:32.560203 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:32.560288 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:32.561123 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:32.561866 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:32.562753 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:32.563165 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:32.563286 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:32.565085 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:32.565089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:32.565092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:32.565116 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:32.566849 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:32.567389 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:32.567392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:32.567394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:32.567476 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:32.567845 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:32.567849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:32.567851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:32.568054 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:32.568197 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:32.568459 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 20:17:32.568552 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:17:32.577096 12550 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:32.578013 12550 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:32.593641 12550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58182c495175466e8d91a66e67d89358: Generated new TSK 0
I20260620 20:17:32.594445 12550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:32.599799 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 12466
W20260620 20:17:32.605639 12451 connection.cc:535] client connection to 127.0.0.1:45239 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45239): Connection reset by peer (error 104)
2026-06-20T20:17:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:32 dist-test-slave-pflr krb5kdc[12565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:32 dist-test-slave-pflr krb5kdc[12565](info): set up 2 sockets
Jun 20 20:17:32 dist-test-slave-pflr krb5kdc[12565](info): commencing operation
krb5kdc: starting...
W20260620 20:17:33.641057 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:33 dist-test-slave-pflr krb5kdc[12565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:17:33.759615 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:33.886855 12578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:33.887105 12578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:33.887147 12578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:33.891139 12578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:33.891194 12578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:33.891213 12578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:33.891230 12578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:33.891274 12578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:33.896382 12578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:33.897459 12578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:33.898442 12578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:33.906848 12584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:33.906874 12583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:33.906852 12586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:33.907163 12578 server_base.cc:1034] running on GCE node
I20260620 20:17:33.907835 12578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:33.908950 12578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:33.910156 12578 hybrid_clock.cc:648] HybridClock initialized: now 1781986653910142 us; error 33 us; skew 500 ppm
Jun 20 20:17:33 dist-test-slave-pflr krb5kdc[12565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:33.913043 12578 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:17:33.913823 12578 webserver.cc:460] Webserver started at http://127.0.0.1:33105/ using document root <none> and password file <none>
I20260620 20:17:33.914331 12578 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:33.914386 12578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:33.914546 12578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:33.916500 12578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "3d61ad484ebc45b987f19d0b455c6466"
format_stamp: "Formatted at 2026-06-20 20:17:33 on dist-test-slave-pflr"
server_key: "d32e1f9b0585fee037afd63a9c0480d3"
server_key_iv: "2a7f6c60dcb563b3f5a85352c0c8576b"
server_key_version: "encryptionkey@0"
I20260620 20:17:33.917086 12578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "3d61ad484ebc45b987f19d0b455c6466"
format_stamp: "Formatted at 2026-06-20 20:17:33 on dist-test-slave-pflr"
server_key: "d32e1f9b0585fee037afd63a9c0480d3"
server_key_iv: "2a7f6c60dcb563b3f5a85352c0c8576b"
server_key_version: "encryptionkey@0"
I20260620 20:17:33.920780 12578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:33.923237 12593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:33.924494 12578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:33.924666 12578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "3d61ad484ebc45b987f19d0b455c6466"
format_stamp: "Formatted at 2026-06-20 20:17:33 on dist-test-slave-pflr"
server_key: "d32e1f9b0585fee037afd63a9c0480d3"
server_key_iv: "2a7f6c60dcb563b3f5a85352c0c8576b"
server_key_version: "encryptionkey@0"
I20260620 20:17:33.924768 12578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:33.949004 12578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:33.952543 12578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:33.952778 12578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:33.960065 12578 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43549
I20260620 20:17:33.960084 12645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43549 every 8 connection(s)
I20260620 20:17:33.961333 12578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:33.963999 12646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:33.965754 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 12578
I20260620 20:17:33.965914 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:33.966236 24658 external_mini_cluster.cc:1428] Setting key f90435b12fafd4ca1d85fc10b62eaaf9
I20260620 20:17:33.970284 12646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466: Bootstrap starting.
I20260620 20:17:33.972381 12646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:33.973163 12646 log.cc:824] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:33.975369 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:33.967647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48197 (local address 127.0.0.1:43549)
0620 20:17:33.968081 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:33.968096 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:33.968131 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:17:33.968841 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:33.968858 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:33.969207 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:33.969530 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:33.969955 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:33.970963 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:33.971850 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:33.972325 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:33.972510 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:33.973429 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:33.973437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:33.973453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:33.973924 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:33.973988 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:33.974019 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:33.974161 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:33.974372 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 20:17:33.974496 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260620 20:17:33.975396 12646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466: No bootstrap required, opened a new log
I20260620 20:17:33.978277 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:33.979313 12646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d61ad484ebc45b987f19d0b455c6466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20260620 20:17:33.979647 12646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:33.979738 12646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d61ad484ebc45b987f19d0b455c6466, State: Initialized, Role: FOLLOWER
I20260620 20:17:33.980194 12646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d61ad484ebc45b987f19d0b455c6466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20260620 20:17:33.980362 12646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:33.980435 12646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:33.980543 12646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:33.981635 12646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d61ad484ebc45b987f19d0b455c6466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20260620 20:17:33.981992 12646 leader_election.cc:302] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d61ad484ebc45b987f19d0b455c6466; no voters:
I20260620 20:17:33.982316 12646 leader_election.cc:288] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:33.982432 12656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:33.982642 12656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [term 1 LEADER]: Becoming Leader. State: Replica: 3d61ad484ebc45b987f19d0b455c6466, State: Running, Role: LEADER
I20260620 20:17:33.982932 12656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d61ad484ebc45b987f19d0b455c6466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } }
I20260620 20:17:33.983544 12646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:33.984513 12658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d61ad484ebc45b987f19d0b455c6466. Latest consensus state: current_term: 1 leader_uuid: "3d61ad484ebc45b987f19d0b455c6466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d61ad484ebc45b987f19d0b455c6466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } } }
I20260620 20:17:33.984951 12658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:33.984848 12657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d61ad484ebc45b987f19d0b455c6466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d61ad484ebc45b987f19d0b455c6466" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43549 } } }
I20260620 20:17:33.985302 12657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:33.985435 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:33.980325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32825 (local address 127.0.0.1:43549)
0620 20:17:33.980528 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:33.980532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:33.980790 (+ 258us) server_negotiation.cc:407] Connection header received
0620 20:17:33.980904 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:33.980906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:33.980945 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:33.981010 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:33.981282 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:33.981880 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:33.983003 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:33.983314 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:33.983428 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:33.984674 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:33.984678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:33.984683 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:33.984698 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:33.984724 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:33.984737 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:33.984840 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:33.985206 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 20:17:33.985247 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
W20260620 20:17:33.988054 12671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:17:33.988138 12671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:17:33.988211 12667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:33.989147 12667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:33.994288 12667 catalog_manager.cc:1347] Generated new cluster ID: 4a4bf8a0c3974f7aa0506118706ae30f
I20260620 20:17:33.994346 12667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:34.000226 12667 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:34.001381 12667 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:34.012743 12667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d61ad484ebc45b987f19d0b455c6466: Generated new TSK 0
I20260620 20:17:34.013406 12667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:34.023993 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 12578
W20260620 20:17:34.029551 12563 connection.cc:535] client connection to 127.0.0.1:43549 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43549): Connection reset by peer (error 104)
2026-06-20T20:17:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:34 dist-test-slave-pflr krb5kdc[12677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:34 dist-test-slave-pflr krb5kdc[12677](info): set up 2 sockets
Jun 20 20:17:34 dist-test-slave-pflr krb5kdc[12677](info): commencing operation
krb5kdc: starting...
W20260620 20:17:35.061076 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:35 dist-test-slave-pflr krb5kdc[12677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:17:35.183271 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:35.292376 12690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:35.292685 12690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:35.292737 12690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:35.296540 12690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:35.296584 12690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:35.296665 12690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:35.296691 12690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:35.296707 12690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:35.301707 12690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:35.302620 12690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:35.303545 12690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:35.311808 12696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:35.311817 12698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:35.311817 12695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:35.311929 12690 server_base.cc:1034] running on GCE node
I20260620 20:17:35.312659 12690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:35.313752 12690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:35.314908 12690 hybrid_clock.cc:648] HybridClock initialized: now 1781986655314889 us; error 35 us; skew 500 ppm
Jun 20 20:17:35 dist-test-slave-pflr krb5kdc[12677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:35.317811 12690 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:17:35.318676 12690 webserver.cc:460] Webserver started at http://127.0.0.1:36259/ using document root <none> and password file <none>
I20260620 20:17:35.319212 12690 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:35.319298 12690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:35.319557 12690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:35.321594 12690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "db4c8cacdc5745d68952012dd5e22615"
format_stamp: "Formatted at 2026-06-20 20:17:35 on dist-test-slave-pflr"
server_key: "8c076b189deffda8b09f16aa46e1b0c2"
server_key_iv: "2d16198b6a9f8298024b6d665a46472f"
server_key_version: "encryptionkey@0"
I20260620 20:17:35.322163 12690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "db4c8cacdc5745d68952012dd5e22615"
format_stamp: "Formatted at 2026-06-20 20:17:35 on dist-test-slave-pflr"
server_key: "8c076b189deffda8b09f16aa46e1b0c2"
server_key_iv: "2d16198b6a9f8298024b6d665a46472f"
server_key_version: "encryptionkey@0"
I20260620 20:17:35.325734 12690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:17:35.328011 12705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:35.329192 12690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:35.329327 12690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "db4c8cacdc5745d68952012dd5e22615"
format_stamp: "Formatted at 2026-06-20 20:17:35 on dist-test-slave-pflr"
server_key: "8c076b189deffda8b09f16aa46e1b0c2"
server_key_iv: "2d16198b6a9f8298024b6d665a46472f"
server_key_version: "encryptionkey@0"
I20260620 20:17:35.329432 12690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:35.333827 12690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:35.337038 12690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:35.337219 12690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:35.343892 12690 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41351
I20260620 20:17:35.343963 12757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41351 every 8 connection(s)
I20260620 20:17:35.344856 12690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:35.347431 12758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:35.348673 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 12690
I20260620 20:17:35.348831 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:35.349136 24658 external_mini_cluster.cc:1428] Setting key a62d4132b7c5d7829ab53c806ccb9ae8
I20260620 20:17:35.353305 12758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615: Bootstrap starting.
I20260620 20:17:35.354202 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:35.350357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53869 (local address 127.0.0.1:41351)
0620 20:17:35.350726 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:35.350746 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:35.350976 (+ 230us) server_negotiation.cc:407] Connection header received
0620 20:17:35.351735 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:35.351751 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:35.352011 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:35.352324 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:35.353062 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:35.353067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:35.353083 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:35.353098 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:35.353145 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:35.353162 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:35.353312 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:35.353381 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 20:17:35.353542 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":126,"threads_started":1}
I20260620 20:17:35.355625 12758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:35.356499 12758 log.cc:824] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:35.357069 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:35.358668 12758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615: No bootstrap required, opened a new log
I20260620 20:17:35.361162 12761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:35.359317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47421 (local address 127.0.0.1:41351)
0620 20:17:35.359438 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:35.359442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:35.359627 (+ 185us) server_negotiation.cc:407] Connection header received
0620 20:17:35.359735 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:35.359737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:35.359764 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:35.359825 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:35.360581 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:35.360584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:35.360586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:35.360654 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:35.360678 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:35.360694 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:35.360768 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:35.360932 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 20:17:35.361014 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:17:35.362867 12758 raft_consensus.cc:348] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4c8cacdc5745d68952012dd5e22615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260620 20:17:35.363070 12758 raft_consensus.cc:374] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:35.363104 12758 raft_consensus.cc:729] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db4c8cacdc5745d68952012dd5e22615, State: Initialized, Role: FOLLOWER
I20260620 20:17:35.363467 12758 consensus_queue.cc:260] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4c8cacdc5745d68952012dd5e22615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260620 20:17:35.363637 12758 raft_consensus.cc:388] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:35.363713 12758 raft_consensus.cc:482] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:35.363828 12758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:35.364738 12758 raft_consensus.cc:504] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4c8cacdc5745d68952012dd5e22615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260620 20:17:35.365069 12758 leader_election.cc:302] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db4c8cacdc5745d68952012dd5e22615; no voters:
I20260620 20:17:35.365334 12758 leader_election.cc:288] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:35.365468 12768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:35.365713 12768 raft_consensus.cc:686] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [term 1 LEADER]: Becoming Leader. State: Replica: db4c8cacdc5745d68952012dd5e22615, State: Running, Role: LEADER
I20260620 20:17:35.366014 12768 consensus_queue.cc:237] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4c8cacdc5745d68952012dd5e22615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } }
I20260620 20:17:35.366466 12758 sys_catalog.cc:564] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:35.367520 12769 sys_catalog.cc:455] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db4c8cacdc5745d68952012dd5e22615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4c8cacdc5745d68952012dd5e22615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } } }
I20260620 20:17:35.367653 12769 sys_catalog.cc:458] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:35.367980 12770 sys_catalog.cc:455] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db4c8cacdc5745d68952012dd5e22615. Latest consensus state: current_term: 1 leader_uuid: "db4c8cacdc5745d68952012dd5e22615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db4c8cacdc5745d68952012dd5e22615" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41351 } } }
I20260620 20:17:35.368064 12777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:35.368083 12770 sys_catalog.cc:458] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:35.370862 12777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:35.376062 12777 catalog_manager.cc:1347] Generated new cluster ID: cf547581d6974a9baf95e3f2c520e119
I20260620 20:17:35.376128 12777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:35.386269 12777 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:35.386554 12777 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:35.393726 12777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db4c8cacdc5745d68952012dd5e22615: Generated new TSK 0
I20260620 20:17:35.394408 12777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:35.403735 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 12690
2026-06-20T20:17:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:35 dist-test-slave-pflr krb5kdc[12789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:35 dist-test-slave-pflr krb5kdc[12789](info): set up 2 sockets
Jun 20 20:17:35 dist-test-slave-pflr krb5kdc[12789](info): commencing operation
krb5kdc: starting...
W20260620 20:17:36.497108 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:36 dist-test-slave-pflr krb5kdc[12789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:17:36.615280 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37673
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:36.720167 12802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:36.720459 12802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:36.720535 12802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:36.724500 12802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:36.724617 12802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:36.724660 12802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:36.724695 12802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:36.724725 12802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:36.729810 12802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:36.730803 12802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:36.731767 12802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:36.740370 12807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:36.740382 12808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:36.740406 12810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:36.740559 12802 server_base.cc:1034] running on GCE node
I20260620 20:17:36.741174 12802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:36.742264 12802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:36.743448 12802 hybrid_clock.cc:648] HybridClock initialized: now 1781986656743429 us; error 34 us; skew 500 ppm
Jun 20 20:17:36 dist-test-slave-pflr krb5kdc[12789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:36.749310 12802 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:17:36.750201 12802 webserver.cc:460] Webserver started at http://127.0.0.1:43149/ using document root <none> and password file <none>
I20260620 20:17:36.750746 12802 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:36.750833 12802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:36.751058 12802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:36.753089 12802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "9d9a42e7b0b848cda38bde023ff42372"
format_stamp: "Formatted at 2026-06-20 20:17:36 on dist-test-slave-pflr"
server_key: "8b14af0b08467159e9ffae99ce20cbca"
server_key_iv: "a7717517aaa5ab7578cf86fdac41bb37"
server_key_version: "encryptionkey@0"
I20260620 20:17:36.753633 12802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "9d9a42e7b0b848cda38bde023ff42372"
format_stamp: "Formatted at 2026-06-20 20:17:36 on dist-test-slave-pflr"
server_key: "8b14af0b08467159e9ffae99ce20cbca"
server_key_iv: "a7717517aaa5ab7578cf86fdac41bb37"
server_key_version: "encryptionkey@0"
I20260620 20:17:36.757193 12802 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:36.759454 12817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:36.760674 12802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:17:36.760802 12802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "9d9a42e7b0b848cda38bde023ff42372"
format_stamp: "Formatted at 2026-06-20 20:17:36 on dist-test-slave-pflr"
server_key: "8b14af0b08467159e9ffae99ce20cbca"
server_key_iv: "a7717517aaa5ab7578cf86fdac41bb37"
server_key_version: "encryptionkey@0"
I20260620 20:17:36.760908 12802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:36.783418 12802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:36.786219 12802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:36.786425 12802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:36.793567 12802 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37673
I20260620 20:17:36.793601 12869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37673 every 8 connection(s)
I20260620 20:17:36.794492 12802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:36.797127 12870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:36.800880 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 12802
I20260620 20:17:36.801025 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:36.801313 24658 external_mini_cluster.cc:1428] Setting key a13e8521226c5b73c3d584b3e40ae1e0
I20260620 20:17:36.802848 12870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372: Bootstrap starting.
I20260620 20:17:36.805596 12870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:36.806356 12870 log.cc:824] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:36.808175 12870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372: No bootstrap required, opened a new log
Jun 20 20:17:36 dist-test-slave-pflr krb5kdc[12789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:17:36.812500 12870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d9a42e7b0b848cda38bde023ff42372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37673 } }
I20260620 20:17:36.812777 12870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:36.812857 12870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d9a42e7b0b848cda38bde023ff42372, State: Initialized, Role: FOLLOWER
I20260620 20:17:36.813335 12870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d9a42e7b0b848cda38bde023ff42372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37673 } }
I20260620 20:17:36.813459 12870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:36.813500 12870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:36.813632 12870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:36.814777 12870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d9a42e7b0b848cda38bde023ff42372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37673 } }
I20260620 20:17:36.815143 12870 leader_election.cc:302] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d9a42e7b0b848cda38bde023ff42372; no voters:
I20260620 20:17:36.815447 12870 leader_election.cc:288] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:36.815563 12875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:36.815781 12875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [term 1 LEADER]: Becoming Leader. State: Replica: 9d9a42e7b0b848cda38bde023ff42372, State: Running, Role: LEADER
I20260620 20:17:36.816090 12875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d9a42e7b0b848cda38bde023ff42372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37673 } }
I20260620 20:17:36.816869 12870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:36.817359 12876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d9a42e7b0b848cda38bde023ff42372. Latest consensus state: current_term: 1 leader_uuid: "9d9a42e7b0b848cda38bde023ff42372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d9a42e7b0b848cda38bde023ff42372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37673 } } }
I20260620 20:17:36.817445 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:36.802746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57531 (local address 127.0.0.1:37673)
0620 20:17:36.803208 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:36.803232 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:36.803274 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:17:36.804139 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:36.804163 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:36.804642 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:36.804997 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:36.805764 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:36.806832 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:36.807716 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:36.808191 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:36.808361 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:36.811873 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:36.811878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:36.811891 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:36.811917 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:36.814010 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:36.814526 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:36.814537 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:36.814542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:36.814663 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:36.815056 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:36.815060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:36.815061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:36.815633 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:36.815849 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:36.816152 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 20:17:36.816454 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":148,"threads_started":1}
I20260620 20:17:36.817512 12876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:36.818159 12877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d9a42e7b0b848cda38bde023ff42372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d9a42e7b0b848cda38bde023ff42372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37673 } } }
I20260620 20:17:36.818250 12877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:36.820719 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 20:17:36.821838 12889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:17:36.821897 12889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:17:36.821981 12891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:36.822760 12891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:36.828733 12891 catalog_manager.cc:1347] Generated new cluster ID: b14d5be4381e450bbeb0e5f7e0a62d30
I20260620 20:17:36.828799 12891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:36.832989 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:36.824622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42917 (local address 127.0.0.1:37673)
0620 20:17:36.824721 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:36.824727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:36.825115 (+ 388us) server_negotiation.cc:407] Connection header received
0620 20:17:36.825270 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:36.825273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:36.825316 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:36.825402 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:36.826072 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:36.826796 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:36.827598 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:36.827855 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:36.827965 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:36.829627 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:36.829632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:36.829635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:36.829663 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:36.831352 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:36.831795 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:36.831799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:36.831800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:36.831888 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:36.832158 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:36.832161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:36.832163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:36.832371 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:36.832505 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:36.832726 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 20:17:36.832842 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 20:17:36.847075 12891 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:36.848331 12891 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:36.857787 12891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d9a42e7b0b848cda38bde023ff42372: Generated new TSK 0
I20260620 20:17:36.858568 12891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:36.866343 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 12802
2026-06-20T20:17:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:36 dist-test-slave-pflr krb5kdc[12901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:36 dist-test-slave-pflr krb5kdc[12901](info): set up 2 sockets
Jun 20 20:17:36 dist-test-slave-pflr krb5kdc[12901](info): commencing operation
krb5kdc: starting...
W20260620 20:17:37.908957 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:37 dist-test-slave-pflr krb5kdc[12901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:17:38.027968 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:38.136171 12914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:38.136440 12914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:38.136513 12914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:38.140390 12914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:38.140481 12914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:38.140520 12914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:38.140579 12914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:38.140632 12914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:38.145722 12914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:38.146677 12914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:38.147632 12914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:38.155956 12920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:38.156004 12919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:38.155958 12922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:38.156236 12914 server_base.cc:1034] running on GCE node
I20260620 20:17:38.156787 12914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:38.157747 12914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:38.158901 12914 hybrid_clock.cc:648] HybridClock initialized: now 1781986658158879 us; error 34 us; skew 500 ppm
Jun 20 20:17:38 dist-test-slave-pflr krb5kdc[12901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:38.161592 12914 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:17:38.162431 12914 webserver.cc:460] Webserver started at http://127.0.0.1:34813/ using document root <none> and password file <none>
I20260620 20:17:38.162985 12914 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:38.163051 12914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:38.163220 12914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:38.165196 12914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "bb0fd06f8dc047d9abaa0017b916b066"
format_stamp: "Formatted at 2026-06-20 20:17:38 on dist-test-slave-pflr"
server_key: "c26196e379012caef8b28b565ed56e57"
server_key_iv: "56aa8b4fe93d90fec77206f2d23f9140"
server_key_version: "encryptionkey@0"
I20260620 20:17:38.165676 12914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "bb0fd06f8dc047d9abaa0017b916b066"
format_stamp: "Formatted at 2026-06-20 20:17:38 on dist-test-slave-pflr"
server_key: "c26196e379012caef8b28b565ed56e57"
server_key_iv: "56aa8b4fe93d90fec77206f2d23f9140"
server_key_version: "encryptionkey@0"
I20260620 20:17:38.169116 12914 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:38.171288 12929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:38.172569 12914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:38.172765 12914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "bb0fd06f8dc047d9abaa0017b916b066"
format_stamp: "Formatted at 2026-06-20 20:17:38 on dist-test-slave-pflr"
server_key: "c26196e379012caef8b28b565ed56e57"
server_key_iv: "56aa8b4fe93d90fec77206f2d23f9140"
server_key_version: "encryptionkey@0"
I20260620 20:17:38.172873 12914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:38.197316 12914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:38.200256 12914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:38.200465 12914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:38.207494 12914 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43913
I20260620 20:17:38.207522 12981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43913 every 8 connection(s)
I20260620 20:17:38.208405 12914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:38.211071 12982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:38.213779 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 12914
I20260620 20:17:38.213948 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:38.214236 24658 external_mini_cluster.cc:1428] Setting key e84bbcc9532b0684d298a17c74ff447d
I20260620 20:17:38.216934 12982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066: Bootstrap starting.
I20260620 20:17:38.219349 12982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:38.220155 12982 log.cc:824] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:38.222262 12982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066: No bootstrap required, opened a new log
I20260620 20:17:38.223097 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:38.215504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34533 (local address 127.0.0.1:43913)
0620 20:17:38.215897 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:38.215911 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:38.215950 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:17:38.216526 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:38.216542 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:38.216933 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:38.217241 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:38.217573 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:38.218506 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:38.219737 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:38.220184 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:38.220354 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:38.221236 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:38.221243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:38.221257 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:38.221718 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:38.221782 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:38.221811 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:38.221906 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:38.222047 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 20:17:38.222233 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":126,"threads_started":1}
I20260620 20:17:38.225759 12982 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0fd06f8dc047d9abaa0017b916b066" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43913 } }
I20260620 20:17:38.226044 12982 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:38.226071 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:38.226132 12982 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0fd06f8dc047d9abaa0017b916b066, State: Initialized, Role: FOLLOWER
I20260620 20:17:38.226575 12982 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0fd06f8dc047d9abaa0017b916b066" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43913 } }
I20260620 20:17:38.226718 12982 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:38.226784 12982 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:38.226878 12982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:38.227811 12982 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0fd06f8dc047d9abaa0017b916b066" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43913 } }
I20260620 20:17:38.228190 12982 leader_election.cc:302] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb0fd06f8dc047d9abaa0017b916b066; no voters:
I20260620 20:17:38.228536 12982 leader_election.cc:288] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:38.228657 12992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:38.228905 12992 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [term 1 LEADER]: Becoming Leader. State: Replica: bb0fd06f8dc047d9abaa0017b916b066, State: Running, Role: LEADER
I20260620 20:17:38.229228 12992 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0fd06f8dc047d9abaa0017b916b066" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43913 } }
I20260620 20:17:38.229768 12982 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:38.230716 12993 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb0fd06f8dc047d9abaa0017b916b066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0fd06f8dc047d9abaa0017b916b066" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43913 } } }
I20260620 20:17:38.230840 12993 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:38.231057 12993 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb0fd06f8dc047d9abaa0017b916b066. Latest consensus state: current_term: 1 leader_uuid: "bb0fd06f8dc047d9abaa0017b916b066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0fd06f8dc047d9abaa0017b916b066" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43913 } } }
I20260620 20:17:38.231156 12993 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:38.231441 13001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:38.233105 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:38.228203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44463 (local address 127.0.0.1:43913)
0620 20:17:38.228324 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:38.228328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:38.228478 (+ 150us) server_negotiation.cc:407] Connection header received
0620 20:17:38.228695 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:38.228698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:38.228750 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:38.228840 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:38.229205 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:38.229900 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:38.231253 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:38.231581 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:38.231655 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:38.232689 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:38.232693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:38.232697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:38.232710 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:38.232738 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:38.232752 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:38.232828 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:38.232934 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 20:17:38.232982 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:17:38.234704 13001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:38.240119 13001 catalog_manager.cc:1347] Generated new cluster ID: 2f9a67fd61b34ae1ae25e46f068edfe6
I20260620 20:17:38.240180 13001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:38.257268 13001 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:38.258143 13001 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:38.265656 13001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb0fd06f8dc047d9abaa0017b916b066: Generated new TSK 0
I20260620 20:17:38.266433 13001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:38.279830 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 12914
2026-06-20T20:17:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:38 dist-test-slave-pflr krb5kdc[13013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:38 dist-test-slave-pflr krb5kdc[13013](info): set up 2 sockets
Jun 20 20:17:38 dist-test-slave-pflr krb5kdc[13013](info): commencing operation
krb5kdc: starting...
W20260620 20:17:39.341030 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:39 dist-test-slave-pflr krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:17:39.459787 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:39.567914 13026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:39.568190 13026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:39.568261 13026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:39.572153 13026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:39.572234 13026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:39.572271 13026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:39.572325 13026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:39.572356 13026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:39.577380 13026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:39.578346 13026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:39.579327 13026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:39.587651 13032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:39.587659 13031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:39.587754 13026 server_base.cc:1034] running on GCE node
W20260620 20:17:39.587659 13034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:39.588428 13026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:39.589490 13026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:39.590653 13026 hybrid_clock.cc:648] HybridClock initialized: now 1781986659590642 us; error 23 us; skew 500 ppm
Jun 20 20:17:39 dist-test-slave-pflr krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:39.593343 13026 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:17:39.594163 13026 webserver.cc:460] Webserver started at http://127.0.0.1:38221/ using document root <none> and password file <none>
I20260620 20:17:39.594712 13026 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:39.594796 13026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:39.595016 13026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:39.597003 13026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "fb622a25b0b94bf5b802d49552904ad0"
format_stamp: "Formatted at 2026-06-20 20:17:39 on dist-test-slave-pflr"
server_key: "8eeb83eb9872a67fee767e2d3716442f"
server_key_iv: "874f305662ecc187c2b4f849a1602194"
server_key_version: "encryptionkey@0"
I20260620 20:17:39.597550 13026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "fb622a25b0b94bf5b802d49552904ad0"
format_stamp: "Formatted at 2026-06-20 20:17:39 on dist-test-slave-pflr"
server_key: "8eeb83eb9872a67fee767e2d3716442f"
server_key_iv: "874f305662ecc187c2b4f849a1602194"
server_key_version: "encryptionkey@0"
I20260620 20:17:39.600999 13026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:39.603246 13041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:39.604347 13026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:39.604475 13026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "fb622a25b0b94bf5b802d49552904ad0"
format_stamp: "Formatted at 2026-06-20 20:17:39 on dist-test-slave-pflr"
server_key: "8eeb83eb9872a67fee767e2d3716442f"
server_key_iv: "874f305662ecc187c2b4f849a1602194"
server_key_version: "encryptionkey@0"
I20260620 20:17:39.604583 13026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:39.608847 13026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:39.611956 13026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:39.612136 13026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:39.618961 13026 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40953
I20260620 20:17:39.618988 13093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40953 every 8 connection(s)
I20260620 20:17:39.619802 13026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:39.622733 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:39.625197 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 13026
I20260620 20:17:39.625327 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:39.625597 24658 external_mini_cluster.cc:1428] Setting key a4c1a9c1b2588c55c45c54071d3c6e05
I20260620 20:17:39.629009 13094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0: Bootstrap starting.
I20260620 20:17:39.630494 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:39.626954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55103 (local address 127.0.0.1:40953)
0620 20:17:39.627383 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:39.627397 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:39.627429 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:17:39.628051 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:39.628066 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:39.628324 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:39.628673 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:39.629427 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:39.629431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:39.629442 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:39.629452 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:39.629500 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:39.629525 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:39.629668 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:39.629737 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 20:17:39.629878 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":137,"threads_started":1}
I20260620 20:17:39.631481 13094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:39.632298 13094 log.cc:824] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:39.633164 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:39.634326 13094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0: No bootstrap required, opened a new log
I20260620 20:17:39.636665 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:39.635084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51557 (local address 127.0.0.1:40953)
0620 20:17:39.635182 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:39.635188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:39.635371 (+ 183us) server_negotiation.cc:407] Connection header received
0620 20:17:39.635496 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:39.635499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:39.635528 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:39.635618 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:39.636285 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:39.636288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:39.636291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:39.636301 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:39.636325 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:39.636336 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:39.636428 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:39.636499 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 20:17:39.636534 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 20:17:39.638645 13094 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb622a25b0b94bf5b802d49552904ad0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40953 } }
I20260620 20:17:39.638863 13094 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:39.638906 13094 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb622a25b0b94bf5b802d49552904ad0, State: Initialized, Role: FOLLOWER
I20260620 20:17:39.639278 13094 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb622a25b0b94bf5b802d49552904ad0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40953 } }
I20260620 20:17:39.639400 13094 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:39.639457 13094 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:39.639528 13094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:39.640363 13094 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb622a25b0b94bf5b802d49552904ad0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40953 } }
I20260620 20:17:39.640722 13094 leader_election.cc:302] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb622a25b0b94bf5b802d49552904ad0; no voters:
I20260620 20:17:39.641009 13094 leader_election.cc:288] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:39.641076 13104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:39.641295 13104 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [term 1 LEADER]: Becoming Leader. State: Replica: fb622a25b0b94bf5b802d49552904ad0, State: Running, Role: LEADER
I20260620 20:17:39.641589 13104 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb622a25b0b94bf5b802d49552904ad0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40953 } }
I20260620 20:17:39.642259 13094 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:39.643085 13106 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb622a25b0b94bf5b802d49552904ad0. Latest consensus state: current_term: 1 leader_uuid: "fb622a25b0b94bf5b802d49552904ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb622a25b0b94bf5b802d49552904ad0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40953 } } }
I20260620 20:17:39.643208 13106 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:39.643671 13105 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb622a25b0b94bf5b802d49552904ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb622a25b0b94bf5b802d49552904ad0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40953 } } }
I20260620 20:17:39.643759 13105 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:39.643838 13112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:39.647004 13112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:39.652736 13112 catalog_manager.cc:1347] Generated new cluster ID: 596abfc5d2f644f1aed58c2a0e6b3799
I20260620 20:17:39.652992 13112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:39.659375 13112 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:39.659653 13112 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:39.664016 13112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb622a25b0b94bf5b802d49552904ad0: Generated new TSK 0
I20260620 20:17:39.664676 13112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:39.672057 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 13026
2026-06-20T20:17:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:39 dist-test-slave-pflr krb5kdc[13125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:39 dist-test-slave-pflr krb5kdc[13125](info): set up 2 sockets
Jun 20 20:17:39 dist-test-slave-pflr krb5kdc[13125](info): commencing operation
krb5kdc: starting...
W20260620 20:17:40.701112 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:40 dist-test-slave-pflr krb5kdc[13125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:17:40.819418 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:40.927979 13138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:40.928259 13138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:40.928332 13138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:40.932246 13138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:40.932331 13138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:40.932368 13138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:40.932423 13138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:40.932461 13138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:40.937562 13138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:40.938558 13138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:40.939525 13138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:40.947726 13143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:40.947731 13144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:40.947778 13138 server_base.cc:1034] running on GCE node
W20260620 20:17:40.947731 13146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:40.948437 13138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:40.949612 13138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:40.950793 13138 hybrid_clock.cc:648] HybridClock initialized: now 1781986660950762 us; error 43 us; skew 500 ppm
Jun 20 20:17:40 dist-test-slave-pflr krb5kdc[13125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:40.953722 13138 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:17:40.954557 13138 webserver.cc:460] Webserver started at http://127.0.0.1:40855/ using document root <none> and password file <none>
I20260620 20:17:40.955106 13138 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:40.955193 13138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:40.955425 13138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:40.957429 13138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "0aba6a8c41b94011b2bf58f414d91d32"
format_stamp: "Formatted at 2026-06-20 20:17:40 on dist-test-slave-pflr"
server_key: "1175c94cd9f50019bc71546930189245"
server_key_iv: "fe23e3b7fd94e42e39a491e381d36f6d"
server_key_version: "encryptionkey@0"
I20260620 20:17:40.957983 13138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "0aba6a8c41b94011b2bf58f414d91d32"
format_stamp: "Formatted at 2026-06-20 20:17:40 on dist-test-slave-pflr"
server_key: "1175c94cd9f50019bc71546930189245"
server_key_iv: "fe23e3b7fd94e42e39a491e381d36f6d"
server_key_version: "encryptionkey@0"
I20260620 20:17:40.961545 13138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:40.963817 13153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:40.965067 13138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:17:40.965195 13138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "0aba6a8c41b94011b2bf58f414d91d32"
format_stamp: "Formatted at 2026-06-20 20:17:40 on dist-test-slave-pflr"
server_key: "1175c94cd9f50019bc71546930189245"
server_key_iv: "fe23e3b7fd94e42e39a491e381d36f6d"
server_key_version: "encryptionkey@0"
I20260620 20:17:40.965302 13138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:40.987903 13138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:40.991215 13138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:40.991456 13138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:40.998452 13138 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34619
I20260620 20:17:40.998509 13205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34619 every 8 connection(s)
I20260620 20:17:40.999370 13138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:41.002074 13206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:41.005313 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 13138
I20260620 20:17:41.005511 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:41.005779 24658 external_mini_cluster.cc:1428] Setting key 3b5fe366f3df2a33965b7e431a32b86f
I20260620 20:17:41.007788 13206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32: Bootstrap starting.
I20260620 20:17:41.010026 13206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:41.010720 13206 log.cc:824] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:41.012419 13206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32: No bootstrap required, opened a new log
Jun 20 20:17:41 dist-test-slave-pflr krb5kdc[13125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:17:41.016563 13206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aba6a8c41b94011b2bf58f414d91d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34619 } }
I20260620 20:17:41.016845 13206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:41.016911 13206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aba6a8c41b94011b2bf58f414d91d32, State: Initialized, Role: FOLLOWER
I20260620 20:17:41.017395 13206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aba6a8c41b94011b2bf58f414d91d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34619 } }
I20260620 20:17:41.017558 13206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:41.017652 13206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:41.017755 13206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:41.018857 13206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aba6a8c41b94011b2bf58f414d91d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34619 } }
I20260620 20:17:41.019208 13206 leader_election.cc:302] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aba6a8c41b94011b2bf58f414d91d32; no voters:
I20260620 20:17:41.019521 13206 leader_election.cc:288] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:41.019629 13211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:41.019917 13211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [term 1 LEADER]: Becoming Leader. State: Replica: 0aba6a8c41b94011b2bf58f414d91d32, State: Running, Role: LEADER
I20260620 20:17:41.020299 13211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aba6a8c41b94011b2bf58f414d91d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34619 } }
I20260620 20:17:41.020692 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:41.007120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:48717 (local address 127.0.0.1:34619)
0620 20:17:41.007565 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:41.007580 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:41.007616 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:17:41.008300 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:41.008319 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:41.008731 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:41.009116 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:41.009918 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:41.010997 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:41.011866 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:41.012339 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:41.012520 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:41.015338 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:41.015344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:41.015360 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:41.015390 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:41.017393 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:41.018019 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:41.018026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:41.018035 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:41.018168 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:41.018471 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:41.018474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:41.018476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:41.018992 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:41.019190 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:41.019440 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 20:17:41.019679 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":135,"threads_started":1}
I20260620 20:17:41.020876 13206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:41.022854 13212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aba6a8c41b94011b2bf58f414d91d32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aba6a8c41b94011b2bf58f414d91d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34619 } } }
I20260620 20:17:41.022465 13211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aba6a8c41b94011b2bf58f414d91d32. Latest consensus state: current_term: 1 leader_uuid: "0aba6a8c41b94011b2bf58f414d91d32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aba6a8c41b94011b2bf58f414d91d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34619 } } }
I20260620 20:17:41.023319 13211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:41.023319 13212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:41.023697 13226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:41.023818 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:41.026767 13226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:41.032177 13226 catalog_manager.cc:1347] Generated new cluster ID: d7f57d4b342842ae89010456832900ed
I20260620 20:17:41.032232 13226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:41.036666 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:41.025951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:50521 (local address 127.0.0.1:34619)
0620 20:17:41.026050 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:41.026054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:41.026266 (+ 212us) server_negotiation.cc:407] Connection header received
0620 20:17:41.026372 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:41.026375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:41.026417 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:41.026497 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:41.027230 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:41.027974 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:41.031183 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:41.031493 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:41.031608 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:41.033251 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:41.033254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:41.033257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:41.033278 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:41.034872 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:41.035395 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:41.035398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:41.035400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:41.035470 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:41.035818 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:41.035821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:41.035823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:41.035990 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:41.036117 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:41.036420 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 20:17:41.036504 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 20:17:41.041019 13226 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:41.042109 13226 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:41.046334 13226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aba6a8c41b94011b2bf58f414d91d32: Generated new TSK 0
I20260620 20:17:41.047009 13226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:41.056221 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 13138
W20260620 20:17:41.061476 13123 connection.cc:535] client connection to 127.0.0.1:34619 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34619): Connection reset by peer (error 104)
2026-06-20T20:17:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:41 dist-test-slave-pflr krb5kdc[13237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:41 dist-test-slave-pflr krb5kdc[13237](info): set up 2 sockets
Jun 20 20:17:41 dist-test-slave-pflr krb5kdc[13237](info): commencing operation
krb5kdc: starting...
W20260620 20:17:42.109265 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:42 dist-test-slave-pflr krb5kdc[13237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:17:42.227516 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:42.332471 13250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:42.332782 13250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:42.332863 13250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:42.336759 13250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:42.336812 13250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:42.336830 13250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:42.336848 13250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:42.336862 13250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:42.341987 13250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:42.342963 13250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:42.343930 13250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:42.352098 13258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:42.352129 13256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:42.352291 13250 server_base.cc:1034] running on GCE node
W20260620 20:17:42.352139 13255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:42.352922 13250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:42.353933 13250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:42.355122 13250 hybrid_clock.cc:648] HybridClock initialized: now 1781986662355096 us; error 38 us; skew 500 ppm
Jun 20 20:17:42 dist-test-slave-pflr krb5kdc[13237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:42.361406 13250 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:17:42.362252 13250 webserver.cc:460] Webserver started at http://127.0.0.1:43995/ using document root <none> and password file <none>
I20260620 20:17:42.362814 13250 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:42.362903 13250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:42.363143 13250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:42.365245 13250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "8abf0e627467466faa5df656c0941df3"
format_stamp: "Formatted at 2026-06-20 20:17:42 on dist-test-slave-pflr"
server_key: "685e2cec0213282e38f2b69983b55457"
server_key_iv: "75d4a2da696473175c065fcabc58754a"
server_key_version: "encryptionkey@0"
I20260620 20:17:42.365797 13250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "8abf0e627467466faa5df656c0941df3"
format_stamp: "Formatted at 2026-06-20 20:17:42 on dist-test-slave-pflr"
server_key: "685e2cec0213282e38f2b69983b55457"
server_key_iv: "75d4a2da696473175c065fcabc58754a"
server_key_version: "encryptionkey@0"
I20260620 20:17:42.369383 13250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:17:42.371636 13265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:42.372828 13250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:17:42.372956 13250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "8abf0e627467466faa5df656c0941df3"
format_stamp: "Formatted at 2026-06-20 20:17:42 on dist-test-slave-pflr"
server_key: "685e2cec0213282e38f2b69983b55457"
server_key_iv: "75d4a2da696473175c065fcabc58754a"
server_key_version: "encryptionkey@0"
I20260620 20:17:42.373068 13250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:42.397821 13250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:42.401108 13250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:42.401309 13250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:42.408115 13250 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40317
I20260620 20:17:42.408138 13317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40317 every 8 connection(s)
I20260620 20:17:42.409087 13250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:42.411720 13318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:42.412921 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 13250
I20260620 20:17:42.413071 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:42.413411 24658 external_mini_cluster.cc:1428] Setting key 427406c62839020412d89cb3a99f7e7d
I20260620 20:17:42.417588 13318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3: Bootstrap starting.
I20260620 20:17:42.419602 13318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:42.420363 13318 log.cc:824] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:42.421911 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:42.414757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:34363 (local address 127.0.0.1:40317)
0620 20:17:42.415148 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:42.415162 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:42.415194 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:17:42.415815 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:42.415831 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:42.416153 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:42.416465 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:42.416888 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:42.417869 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:42.418751 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:42.419204 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:42.419378 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:42.420231 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:42.420241 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:42.420252 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:42.420743 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:42.420796 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:42.420819 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:42.420955 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:42.421142 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 20:17:42.421268 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260620 20:17:42.422277 13318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3: No bootstrap required, opened a new log
I20260620 20:17:42.424769 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:42.426316 13318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abf0e627467466faa5df656c0941df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40317 } }
I20260620 20:17:42.426545 13318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:42.426615 13318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8abf0e627467466faa5df656c0941df3, State: Initialized, Role: FOLLOWER
I20260620 20:17:42.427026 13318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abf0e627467466faa5df656c0941df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40317 } }
I20260620 20:17:42.427173 13318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:42.427258 13318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:42.427361 13318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:42.428164 13318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abf0e627467466faa5df656c0941df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40317 } }
I20260620 20:17:42.428483 13318 leader_election.cc:302] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8abf0e627467466faa5df656c0941df3; no voters:
I20260620 20:17:42.428781 13318 leader_election.cc:288] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:42.428884 13328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:42.429144 13328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [term 1 LEADER]: Becoming Leader. State: Replica: 8abf0e627467466faa5df656c0941df3, State: Running, Role: LEADER
I20260620 20:17:42.429468 13328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abf0e627467466faa5df656c0941df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40317 } }
I20260620 20:17:42.429952 13318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:42.431027 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:42.426684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:49367 (local address 127.0.0.1:40317)
0620 20:17:42.426799 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:42.426803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:42.427018 (+ 215us) server_negotiation.cc:407] Connection header received
0620 20:17:42.427146 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:42.427149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:42.427193 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:42.427262 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:42.427516 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:42.428016 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:42.429046 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:42.429364 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:42.429438 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:42.430466 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:42.430472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:42.430478 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:42.430494 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:42.430532 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:42.430545 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:42.430611 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:42.430840 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 20:17:42.430871 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 20:17:42.431115 13329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8abf0e627467466faa5df656c0941df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abf0e627467466faa5df656c0941df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40317 } } }
I20260620 20:17:42.431228 13329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:42.431550 13330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8abf0e627467466faa5df656c0941df3. Latest consensus state: current_term: 1 leader_uuid: "8abf0e627467466faa5df656c0941df3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abf0e627467466faa5df656c0941df3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40317 } } }
I20260620 20:17:42.431644 13330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:42.431669 13337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:42.434532 13337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:42.439610 13337 catalog_manager.cc:1347] Generated new cluster ID: fc7e4b9cb1ef47c39324c18f214c701c
I20260620 20:17:42.439673 13337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:42.449376 13337 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:42.450474 13337 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:42.458761 13337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8abf0e627467466faa5df656c0941df3: Generated new TSK 0
I20260620 20:17:42.459436 13337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:42.464350 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 13250
W20260620 20:17:42.469401 13235 connection.cc:535] client connection to 127.0.0.1:40317 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40317): Connection reset by peer (error 104)
2026-06-20T20:17:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:42 dist-test-slave-pflr krb5kdc[13349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:42 dist-test-slave-pflr krb5kdc[13349](info): set up 2 sockets
Jun 20 20:17:42 dist-test-slave-pflr krb5kdc[13349](info): commencing operation
krb5kdc: starting...
W20260620 20:17:43.509140 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:43 dist-test-slave-pflr krb5kdc[13349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:17:43.627461 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:43.732851 13362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:43.733081 13362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:43.733121 13362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:43.737068 13362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:43.737115 13362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:43.737134 13362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:43.737174 13362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:43.737206 13362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:43.742290 13362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:43.743237 13362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:43.744176 13362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:43.752203 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:43.752249 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:43.752331 13367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:43.752472 13362 server_base.cc:1034] running on GCE node
I20260620 20:17:43.753032 13362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:43.754094 13362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:43.755282 13362 hybrid_clock.cc:648] HybridClock initialized: now 1781986663755261 us; error 32 us; skew 500 ppm
Jun 20 20:17:43 dist-test-slave-pflr krb5kdc[13349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:43.758054 13362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:17:43.758817 13362 webserver.cc:460] Webserver started at http://127.0.0.1:33303/ using document root <none> and password file <none>
I20260620 20:17:43.759303 13362 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:43.759362 13362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:43.759518 13362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:43.761468 13362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "9eacdb8b604f4a018c891036c49c1b9f"
format_stamp: "Formatted at 2026-06-20 20:17:43 on dist-test-slave-pflr"
server_key: "72a1b9dc1635234841fb094c92403416"
server_key_iv: "f1934a216db1443e4ec90aa81077e74e"
server_key_version: "encryptionkey@0"
I20260620 20:17:43.761972 13362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "9eacdb8b604f4a018c891036c49c1b9f"
format_stamp: "Formatted at 2026-06-20 20:17:43 on dist-test-slave-pflr"
server_key: "72a1b9dc1635234841fb094c92403416"
server_key_iv: "f1934a216db1443e4ec90aa81077e74e"
server_key_version: "encryptionkey@0"
I20260620 20:17:43.765486 13362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:17:43.767683 13377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:43.768889 13362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:17:43.768985 13362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "9eacdb8b604f4a018c891036c49c1b9f"
format_stamp: "Formatted at 2026-06-20 20:17:43 on dist-test-slave-pflr"
server_key: "72a1b9dc1635234841fb094c92403416"
server_key_iv: "f1934a216db1443e4ec90aa81077e74e"
server_key_version: "encryptionkey@0"
I20260620 20:17:43.769063 13362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:43.773427 13362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:43.776468 13362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:43.776661 13362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:43.783203 13362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35861
I20260620 20:17:43.783262 13429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35861 every 8 connection(s)
I20260620 20:17:43.784145 13362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:43.786711 13430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:43.791853 13430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f: Bootstrap starting.
I20260620 20:17:43.792969 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 13362
I20260620 20:17:43.793107 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:43.793385 24658 external_mini_cluster.cc:1428] Setting key 588b93f63c1f09626bd12366b86a1e3c
I20260620 20:17:43.794520 13430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:43.795397 13430 log.cc:824] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:43.797528 13430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f: No bootstrap required, opened a new log
I20260620 20:17:43.798725 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:43.794444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:44999 (local address 127.0.0.1:35861)
0620 20:17:43.794905 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:43.794922 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:43.794961 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:17:43.795836 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:43.795854 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:43.796163 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:43.796523 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:43.797342 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:43.797348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:43.797362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:43.797374 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:43.797442 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:43.797469 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:43.797589 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:43.797732 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 20:17:43.797887 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":135,"threads_started":1}
I20260620 20:17:43.801688 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:43.801812 13430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eacdb8b604f4a018c891036c49c1b9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } }
I20260620 20:17:43.802052 13430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:43.802124 13430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eacdb8b604f4a018c891036c49c1b9f, State: Initialized, Role: FOLLOWER
I20260620 20:17:43.802536 13430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eacdb8b604f4a018c891036c49c1b9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } }
I20260620 20:17:43.802647 13430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:43.802729 13430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:43.802846 13430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:43.803794 13430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eacdb8b604f4a018c891036c49c1b9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } }
I20260620 20:17:43.804180 13430 leader_election.cc:302] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9eacdb8b604f4a018c891036c49c1b9f; no voters:
I20260620 20:17:43.804538 13430 leader_election.cc:288] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:43.804651 13440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:43.804859 13440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [term 1 LEADER]: Becoming Leader. State: Replica: 9eacdb8b604f4a018c891036c49c1b9f, State: Running, Role: LEADER
I20260620 20:17:43.805545 13440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eacdb8b604f4a018c891036c49c1b9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } }
I20260620 20:17:43.805744 13430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:43.806022 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:43.804065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:51529 (local address 127.0.0.1:35861)
0620 20:17:43.804196 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:43.804201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:43.804418 (+ 217us) server_negotiation.cc:407] Connection header received
0620 20:17:43.804548 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:43.804550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:43.804589 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:43.804692 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:43.805553 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:43.805556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:43.805559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:43.805570 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:43.805596 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:43.805608 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:43.805691 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:43.805879 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 20:17:43.805918 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:17:43.806857 13440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eacdb8b604f4a018c891036c49c1b9f. Latest consensus state: current_term: 1 leader_uuid: "9eacdb8b604f4a018c891036c49c1b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eacdb8b604f4a018c891036c49c1b9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } } }
I20260620 20:17:43.806985 13440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:43.807272 13441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9eacdb8b604f4a018c891036c49c1b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eacdb8b604f4a018c891036c49c1b9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } } }
I20260620 20:17:43.807333 13441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f [sys.catalog]: This master's current role is: LEADER
W20260620 20:17:43.810554 13455 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:17:43.810631 13455 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:17:43.810707 13449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:43.811546 13449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:43.816972 13449 catalog_manager.cc:1347] Generated new cluster ID: 3f8fc77b13a84a9d80fd16684161bc14
I20260620 20:17:43.817039 13449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:43.835233 13449 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:43.835654 13449 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:43.849478 13449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eacdb8b604f4a018c891036c49c1b9f: Generated new TSK 0
I20260620 20:17:43.850466 13449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:43.855813 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 13362
2026-06-20T20:17:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:43 dist-test-slave-pflr krb5kdc[13461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:43 dist-test-slave-pflr krb5kdc[13461](info): set up 2 sockets
Jun 20 20:17:43 dist-test-slave-pflr krb5kdc[13461](info): commencing operation
krb5kdc: starting...
W20260620 20:17:44.889173 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:44 dist-test-slave-pflr krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:17:45.009986 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:45.118624 13474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:45.118857 13474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:45.118923 13474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:45.122823 13474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:45.122871 13474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:45.122890 13474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:45.122936 13474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:45.122977 13474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:45.128059 13474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:45.128995 13474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:45.129907 13474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:45.138198 13480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:45.138167 13479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:45.138238 13474 server_base.cc:1034] running on GCE node
W20260620 20:17:45.138167 13482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:45.138912 13474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:45.139941 13474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:45.141094 13474 hybrid_clock.cc:648] HybridClock initialized: now 1781986665141076 us; error 29 us; skew 500 ppm
Jun 20 20:17:45 dist-test-slave-pflr krb5kdc[13461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:45.143684 13474 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:17:45.144439 13474 webserver.cc:460] Webserver started at http://127.0.0.1:36011/ using document root <none> and password file <none>
I20260620 20:17:45.144994 13474 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:45.145056 13474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:45.145220 13474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:45.147110 13474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "cb657b35b13f43af8e559c4b029a42c0"
format_stamp: "Formatted at 2026-06-20 20:17:45 on dist-test-slave-pflr"
server_key: "e7b82d1fd710f4a6976942010d338561"
server_key_iv: "810af5452d2640de717deddf7b9e795b"
server_key_version: "encryptionkey@0"
I20260620 20:17:45.147603 13474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "cb657b35b13f43af8e559c4b029a42c0"
format_stamp: "Formatted at 2026-06-20 20:17:45 on dist-test-slave-pflr"
server_key: "e7b82d1fd710f4a6976942010d338561"
server_key_iv: "810af5452d2640de717deddf7b9e795b"
server_key_version: "encryptionkey@0"
I20260620 20:17:45.151048 13474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:17:45.153203 13489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:45.154382 13474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:17:45.154520 13474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "cb657b35b13f43af8e559c4b029a42c0"
format_stamp: "Formatted at 2026-06-20 20:17:45 on dist-test-slave-pflr"
server_key: "e7b82d1fd710f4a6976942010d338561"
server_key_iv: "810af5452d2640de717deddf7b9e795b"
server_key_version: "encryptionkey@0"
I20260620 20:17:45.154624 13474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:45.168506 13474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:45.171116 13474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:45.171296 13474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:45.177886 13474 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42787
I20260620 20:17:45.177970 13542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42787 every 8 connection(s)
I20260620 20:17:45.178810 13474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:45.181454 13543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:45.185575 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 13474
I20260620 20:17:45.185709 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:45.186025 24658 external_mini_cluster.cc:1428] Setting key cd920735fd3ade8cbd43682b2719af4b
I20260620 20:17:45.187094 13543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0: Bootstrap starting.
I20260620 20:17:45.189293 13543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:45.189989 13543 log.cc:824] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:45.191711 13543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0: No bootstrap required, opened a new log
Jun 20 20:17:45 dist-test-slave-pflr krb5kdc[13461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:17:45.195145 13543 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb657b35b13f43af8e559c4b029a42c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260620 20:17:45.195370 13543 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:45.195454 13543 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb657b35b13f43af8e559c4b029a42c0, State: Initialized, Role: FOLLOWER
I20260620 20:17:45.195888 13543 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb657b35b13f43af8e559c4b029a42c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260620 20:17:45.196039 13543 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:45.196113 13543 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:45.196218 13543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:45.197109 13543 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb657b35b13f43af8e559c4b029a42c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260620 20:17:45.197428 13543 leader_election.cc:302] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb657b35b13f43af8e559c4b029a42c0; no voters:
I20260620 20:17:45.197736 13543 leader_election.cc:288] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:45.197865 13548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:45.198074 13548 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [term 1 LEADER]: Becoming Leader. State: Replica: cb657b35b13f43af8e559c4b029a42c0, State: Running, Role: LEADER
I20260620 20:17:45.198423 13548 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb657b35b13f43af8e559c4b029a42c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } }
I20260620 20:17:45.199028 13543 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:45.200030 13549 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb657b35b13f43af8e559c4b029a42c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb657b35b13f43af8e559c4b029a42c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } } }
I20260620 20:17:45.200150 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:45.187363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:36799 (local address 127.0.0.1:42787)
0620 20:17:45.187772 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:45.187787 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:45.187821 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:17:45.188429 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:45.188445 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:45.188838 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:45.189165 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:45.189955 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:45.190949 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:45.191842 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:45.192305 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:45.192482 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:45.194970 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:45.194975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:45.194986 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:45.195017 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:45.196876 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:45.197391 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:45.197396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:45.197400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:45.197502 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:45.197903 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:45.197906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:45.197908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:45.198406 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:45.198625 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:45.198959 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 20:17:45.199172 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":139,"threads_started":1}
I20260620 20:17:45.200138 13549 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:45.200301 13548 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb657b35b13f43af8e559c4b029a42c0. Latest consensus state: current_term: 1 leader_uuid: "cb657b35b13f43af8e559c4b029a42c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb657b35b13f43af8e559c4b029a42c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42787 } } }
I20260620 20:17:45.200714 13548 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:45.201289 13560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:45.203469 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:45.204442 13560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:45.210242 13560 catalog_manager.cc:1347] Generated new cluster ID: 495af853fa80479f9f6ef12dd617b344
I20260620 20:17:45.211100 13560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:45.214488 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:45.205384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:36499 (local address 127.0.0.1:42787)
0620 20:17:45.205483 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:45.205487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:45.205658 (+ 171us) server_negotiation.cc:407] Connection header received
0620 20:17:45.205770 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:45.205774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:45.205813 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:45.205907 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:45.206568 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:45.207374 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:45.208425 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:45.209452 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:45.209566 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:45.211069 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:45.211074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:45.211079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:45.211106 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:45.212744 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:45.213286 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:45.213289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:45.213291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:45.213369 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:45.213700 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:45.213703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:45.213705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:45.213879 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:45.214007 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:45.214270 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 20:17:45.214353 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 20:17:45.227996 13560 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:45.229010 13560 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:45.241739 13560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb657b35b13f43af8e559c4b029a42c0: Generated new TSK 0
I20260620 20:17:45.242411 13560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:45.250480 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 13474
W20260620 20:17:45.255612 13459 connection.cc:535] client connection to 127.0.0.1:42787 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42787): Connection reset by peer (error 104)
2026-06-20T20:17:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:45 dist-test-slave-pflr krb5kdc[13574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:45 dist-test-slave-pflr krb5kdc[13574](info): set up 2 sockets
Jun 20 20:17:45 dist-test-slave-pflr krb5kdc[13574](info): commencing operation
krb5kdc: starting...
W20260620 20:17:46.341059 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:46 dist-test-slave-pflr krb5kdc[13574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:17:46.460180 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:46.569797 13587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:46.570045 13587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:46.570086 13587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:46.573881 13587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:46.573930 13587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:46.573949 13587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:46.573966 13587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:46.573980 13587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:46.579649 13587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:46.580560 13587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:46.581539 13587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:46.589656 13595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:46.589656 13592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:46.589838 13587 server_base.cc:1034] running on GCE node
W20260620 20:17:46.589620 13593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:46.590509 13587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:46.591543 13587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:46.592664 13587 hybrid_clock.cc:648] HybridClock initialized: now 1781986666592654 us; error 27 us; skew 500 ppm
Jun 20 20:17:46 dist-test-slave-pflr krb5kdc[13574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:46.595528 13587 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:17:46.596449 13587 webserver.cc:460] Webserver started at http://127.0.0.1:39447/ using document root <none> and password file <none>
I20260620 20:17:46.597054 13587 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:46.597143 13587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:46.597366 13587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:46.599334 13587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "1f49510556f54a548885a882aeb6f592"
format_stamp: "Formatted at 2026-06-20 20:17:46 on dist-test-slave-pflr"
server_key: "2aeae1966aaf32f4a108d59c0ecb12ca"
server_key_iv: "ea35939ce83956fe0583caca6857bb1c"
server_key_version: "encryptionkey@0"
I20260620 20:17:46.599884 13587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "1f49510556f54a548885a882aeb6f592"
format_stamp: "Formatted at 2026-06-20 20:17:46 on dist-test-slave-pflr"
server_key: "2aeae1966aaf32f4a108d59c0ecb12ca"
server_key_iv: "ea35939ce83956fe0583caca6857bb1c"
server_key_version: "encryptionkey@0"
I20260620 20:17:46.603500 13587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:17:46.605860 13602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:46.607089 13587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:17:46.607236 13587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "1f49510556f54a548885a882aeb6f592"
format_stamp: "Formatted at 2026-06-20 20:17:46 on dist-test-slave-pflr"
server_key: "2aeae1966aaf32f4a108d59c0ecb12ca"
server_key_iv: "ea35939ce83956fe0583caca6857bb1c"
server_key_version: "encryptionkey@0"
I20260620 20:17:46.607348 13587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:46.620498 13587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:46.623673 13587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:46.623875 13587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:46.630748 13587 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33531
I20260620 20:17:46.630769 13654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33531 every 8 connection(s)
I20260620 20:17:46.631624 13587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:46.634495 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:46.636534 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 13587
I20260620 20:17:46.636685 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:46.636984 24658 external_mini_cluster.cc:1428] Setting key 00c0cbbc408518de8b22ffb624e138e0
I20260620 20:17:46.640988 13655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592: Bootstrap starting.
I20260620 20:17:46.643093 13655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:46.643788 13655 log.cc:824] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:46.645762 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:46.638343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:39403 (local address 127.0.0.1:33531)
0620 20:17:46.638759 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:46.638774 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:46.638808 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:17:46.639428 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:46.639445 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:46.639772 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:46.640099 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:46.640486 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:46.641553 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:46.642469 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:46.642937 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:46.643111 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:46.643994 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:46.643999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:46.644011 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:46.644373 (+ 362us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:46.644424 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:46.644445 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:46.644622 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:46.644760 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 20:17:46.644893 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":122,"threads_started":1}
I20260620 20:17:46.645977 13655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592: No bootstrap required, opened a new log
I20260620 20:17:46.648769 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:46.649823 13655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f49510556f54a548885a882aeb6f592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260620 20:17:46.650130 13655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:46.650220 13655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f49510556f54a548885a882aeb6f592, State: Initialized, Role: FOLLOWER
I20260620 20:17:46.650664 13655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f49510556f54a548885a882aeb6f592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260620 20:17:46.650828 13655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:46.650892 13655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:46.651008 13655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:46.652040 13655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f49510556f54a548885a882aeb6f592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260620 20:17:46.652403 13655 leader_election.cc:302] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f49510556f54a548885a882aeb6f592; no voters:
I20260620 20:17:46.652729 13655 leader_election.cc:288] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:46.652832 13665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:46.653091 13665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [term 1 LEADER]: Becoming Leader. State: Replica: 1f49510556f54a548885a882aeb6f592, State: Running, Role: LEADER
I20260620 20:17:46.653381 13665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f49510556f54a548885a882aeb6f592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260620 20:17:46.653968 13655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:46.654443 13665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f49510556f54a548885a882aeb6f592. Latest consensus state: current_term: 1 leader_uuid: "1f49510556f54a548885a882aeb6f592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f49510556f54a548885a882aeb6f592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } } }
I20260620 20:17:46.654583 13665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:46.654894 13667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f49510556f54a548885a882aeb6f592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f49510556f54a548885a882aeb6f592" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } } }
I20260620 20:17:46.655010 13667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:46.655988 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:46.650754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:46849 (local address 127.0.0.1:33531)
0620 20:17:46.650876 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:46.650880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:46.651048 (+ 168us) server_negotiation.cc:407] Connection header received
0620 20:17:46.651189 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:46.651191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:46.651227 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:46.651289 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:46.651587 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:46.652087 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:46.653277 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:46.653614 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:46.653691 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:46.655398 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:46.655402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:46.655405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:46.655417 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:46.655444 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:46.655456 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:46.655563 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:46.655805 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 20:17:46.655846 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 20:17:46.659201 13680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:17:46.659265 13680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:17:46.659344 13670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:46.660071 13670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:46.665565 13670 catalog_manager.cc:1347] Generated new cluster ID: d3137696c84441fe90829945cec5604e
I20260620 20:17:46.665625 13670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:46.676450 13670 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:46.677611 13670 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:46.681900 13670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f49510556f54a548885a882aeb6f592: Generated new TSK 0
I20260620 20:17:46.682754 13670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:46.692317 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 13587
W20260620 20:17:46.697736 13572 connection.cc:535] client connection to 127.0.0.1:33531 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33531): Connection reset by peer (error 104)
2026-06-20T20:17:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:46 dist-test-slave-pflr krb5kdc[13686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:46 dist-test-slave-pflr krb5kdc[13686](info): set up 2 sockets
Jun 20 20:17:46 dist-test-slave-pflr krb5kdc[13686](info): commencing operation
krb5kdc: starting...
W20260620 20:17:47.753103 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:47 dist-test-slave-pflr krb5kdc[13686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:17:47.872334 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:47.978631 13699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:47.978919 13699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:47.978997 13699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:47.982955 13699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:47.983044 13699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:47.983083 13699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:47.983140 13699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:47.983178 13699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:47.988209 13699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38967
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:47.989233 13699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:47.990204 13699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:47.998519 13705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:47.998656 13707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:47.998555 13704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:47.998597 13699 server_base.cc:1034] running on GCE node
I20260620 20:17:47.999217 13699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:48.000353 13699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:48.001531 13699 hybrid_clock.cc:648] HybridClock initialized: now 1781986668001502 us; error 38 us; skew 500 ppm
Jun 20 20:17:48 dist-test-slave-pflr krb5kdc[13686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:48.008205 13699 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:17:48.009204 13699 webserver.cc:460] Webserver started at http://127.0.0.1:37615/ using document root <none> and password file <none>
I20260620 20:17:48.009774 13699 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:48.009864 13699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:48.010092 13699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:48.012183 13699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "12c93ac3cfd743c79db317ce82c6ec02"
format_stamp: "Formatted at 2026-06-20 20:17:48 on dist-test-slave-pflr"
server_key: "75980c8b751cf89e16722c9e53efc908"
server_key_iv: "62e77fa5b3c2f4da784ad9cf9c40c7fa"
server_key_version: "encryptionkey@0"
I20260620 20:17:48.012802 13699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "12c93ac3cfd743c79db317ce82c6ec02"
format_stamp: "Formatted at 2026-06-20 20:17:48 on dist-test-slave-pflr"
server_key: "75980c8b751cf89e16722c9e53efc908"
server_key_iv: "62e77fa5b3c2f4da784ad9cf9c40c7fa"
server_key_version: "encryptionkey@0"
I20260620 20:17:48.016278 13699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:17:48.018738 13714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:48.019874 13699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 20:17:48.020020 13699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "12c93ac3cfd743c79db317ce82c6ec02"
format_stamp: "Formatted at 2026-06-20 20:17:48 on dist-test-slave-pflr"
server_key: "75980c8b751cf89e16722c9e53efc908"
server_key_iv: "62e77fa5b3c2f4da784ad9cf9c40c7fa"
server_key_version: "encryptionkey@0"
I20260620 20:17:48.020124 13699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:48.024501 13699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:48.027146 13699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:48.027330 13699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:48.034027 13699 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38967
I20260620 20:17:48.034055 13766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38967 every 8 connection(s)
I20260620 20:17:48.034855 13699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:48.037410 13767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:48.037717 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 13699
I20260620 20:17:48.037858 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:48.038151 24658 external_mini_cluster.cc:1428] Setting key 5fb226a15f36d2b43c5806b479c5e322
I20260620 20:17:48.043329 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:48.039431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:44053 (local address 127.0.0.1:38967)
0620 20:17:48.039804 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:48.039827 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:48.039868 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:17:48.040578 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:48.040601 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:48.040955 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:48.041318 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:48.042003 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:48.042009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:48.042023 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:48.042035 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:48.042092 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:48.042118 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:48.042249 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:48.042325 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 20:17:48.042490 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
I20260620 20:17:48.044216 13767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02: Bootstrap starting.
I20260620 20:17:48.045969 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:48.046525 13767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:48.047358 13767 log.cc:824] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:48.049823 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:48.048256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:37409 (local address 127.0.0.1:38967)
0620 20:17:48.048363 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:48.048367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:48.048514 (+ 147us) server_negotiation.cc:407] Connection header received
0620 20:17:48.048676 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:48.048678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:48.048703 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:48.048794 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:48.049474 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:48.049477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:48.049479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:48.049491 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:48.049530 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:48.049542 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:48.049629 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:48.049691 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 20:17:48.049725 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 20:17:48.049971 13767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02: No bootstrap required, opened a new log
I20260620 20:17:48.053206 13767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c93ac3cfd743c79db317ce82c6ec02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } }
I20260620 20:17:48.053424 13767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:48.053489 13767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c93ac3cfd743c79db317ce82c6ec02, State: Initialized, Role: FOLLOWER
I20260620 20:17:48.053862 13767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c93ac3cfd743c79db317ce82c6ec02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } }
I20260620 20:17:48.054010 13767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:48.054090 13767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:48.054184 13767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:48.055065 13767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c93ac3cfd743c79db317ce82c6ec02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } }
I20260620 20:17:48.055377 13767 leader_election.cc:302] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12c93ac3cfd743c79db317ce82c6ec02; no voters:
I20260620 20:17:48.055696 13767 leader_election.cc:288] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:48.055788 13777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:48.056021 13777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [term 1 LEADER]: Becoming Leader. State: Replica: 12c93ac3cfd743c79db317ce82c6ec02, State: Running, Role: LEADER
I20260620 20:17:48.056349 13777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c93ac3cfd743c79db317ce82c6ec02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } }
I20260620 20:17:48.056849 13767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:48.057963 13778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12c93ac3cfd743c79db317ce82c6ec02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c93ac3cfd743c79db317ce82c6ec02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } } }
I20260620 20:17:48.058084 13778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:48.058473 13786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:48.058415 13779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c93ac3cfd743c79db317ce82c6ec02. Latest consensus state: current_term: 1 leader_uuid: "12c93ac3cfd743c79db317ce82c6ec02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c93ac3cfd743c79db317ce82c6ec02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38967 } } }
I20260620 20:17:48.060673 13779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:48.061369 13786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:48.066499 13786 catalog_manager.cc:1347] Generated new cluster ID: e4267c13e1d74e7690fcf3450ae2bb6f
I20260620 20:17:48.066572 13786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:48.077059 13786 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:48.077340 13786 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:48.085193 13786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12c93ac3cfd743c79db317ce82c6ec02: Generated new TSK 0
I20260620 20:17:48.085927 13786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:48.093510 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 13699
2026-06-20T20:17:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:48 dist-test-slave-pflr krb5kdc[13798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:48 dist-test-slave-pflr krb5kdc[13798](info): set up 2 sockets
Jun 20 20:17:48 dist-test-slave-pflr krb5kdc[13798](info): commencing operation
krb5kdc: starting...
W20260620 20:17:49.129109 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:49 dist-test-slave-pflr krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:17:49.247895 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:45275
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:39719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:45275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:49.355319 13811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:49.355599 13811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:49.355680 13811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:49.359614 13811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:49.359691 13811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:49.359728 13811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:49.359783 13811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:49.359819 13811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:49.364893 13811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:39719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:45275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:45275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:49.365864 13811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:49.366825 13811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:49.375073 13819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:49.375099 13817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:49.375077 13816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:49.375162 13811 server_base.cc:1034] running on GCE node
I20260620 20:17:49.375937 13811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:49.377055 13811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:49.378235 13811 hybrid_clock.cc:648] HybridClock initialized: now 1781986669378201 us; error 48 us; skew 500 ppm
Jun 20 20:17:49 dist-test-slave-pflr krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986669, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:49.380882 13811 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:17:49.381708 13811 webserver.cc:460] Webserver started at http://127.24.20.190:37399/ using document root <none> and password file <none>
I20260620 20:17:49.382242 13811 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:49.382330 13811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:49.382560 13811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:49.384459 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "18d4922401a54c36ba652e1f0330a41f"
format_stamp: "Formatted at 2026-06-20 20:17:49 on dist-test-slave-pflr"
server_key: "a2404806f0b787d81b9b271dcba628c7"
server_key_iv: "14955532b5b5441f75b81ffc967c4097"
server_key_version: "encryptionkey@0"
I20260620 20:17:49.385066 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "18d4922401a54c36ba652e1f0330a41f"
format_stamp: "Formatted at 2026-06-20 20:17:49 on dist-test-slave-pflr"
server_key: "a2404806f0b787d81b9b271dcba628c7"
server_key_iv: "14955532b5b5441f75b81ffc967c4097"
server_key_version: "encryptionkey@0"
I20260620 20:17:49.388499 13811 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:49.390715 13826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:49.391974 13811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:49.392113 13811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "18d4922401a54c36ba652e1f0330a41f"
format_stamp: "Formatted at 2026-06-20 20:17:49 on dist-test-slave-pflr"
server_key: "a2404806f0b787d81b9b271dcba628c7"
server_key_iv: "14955532b5b5441f75b81ffc967c4097"
server_key_version: "encryptionkey@0"
I20260620 20:17:49.392216 13811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:49.407166 13811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:49.410310 13811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:49.410506 13811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:49.417541 13811 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:45275
I20260620 20:17:49.417567 13878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:45275 every 8 connection(s)
I20260620 20:17:49.418419 13811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:49.421032 13879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:49.423437 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 13811
I20260620 20:17:49.423575 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:49.423849 24658 external_mini_cluster.cc:1428] Setting key 886a622cda9dadf231b10d37e18c02ed
I20260620 20:17:49.426728 13879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f: Bootstrap starting.
I20260620 20:17:49.428849 13879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:49.429554 13879 log.cc:824] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:49.431322 13879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f: No bootstrap required, opened a new log
Jun 20 20:17:49 dist-test-slave-pflr krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:17:49.435192 13879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d4922401a54c36ba652e1f0330a41f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 45275 } }
I20260620 20:17:49.435392 13879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:49.435498 13879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d4922401a54c36ba652e1f0330a41f, State: Initialized, Role: FOLLOWER
I20260620 20:17:49.435967 13879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d4922401a54c36ba652e1f0330a41f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 45275 } }
I20260620 20:17:49.436091 13879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:49.436175 13879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:49.436285 13879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:49.437242 13879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d4922401a54c36ba652e1f0330a41f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 45275 } }
I20260620 20:17:49.437583 13879 leader_election.cc:302] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18d4922401a54c36ba652e1f0330a41f; no voters:
I20260620 20:17:49.437863 13879 leader_election.cc:288] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:49.437984 13884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:49.438027 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:49.425183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57565 (local address 127.24.20.190:45275)
0620 20:17:49.425582 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:49.425596 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:49.425629 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:17:49.426211 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:49.426225 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:49.426549 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:49.426866 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:49.427672 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:49.428704 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:49.429606 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:49.430295 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:49.430472 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:49.433150 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:49.433156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:49.433175 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:49.433201 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:49.434880 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:49.435379 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:49.435385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:49.435393 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:49.435595 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:49.435928 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:49.435931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:49.435933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:49.436426 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:49.436698 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:49.436957 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 20:17:49.437170 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260620 20:17:49.438227 13884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [term 1 LEADER]: Becoming Leader. State: Replica: 18d4922401a54c36ba652e1f0330a41f, State: Running, Role: LEADER
I20260620 20:17:49.438552 13884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d4922401a54c36ba652e1f0330a41f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 45275 } }
I20260620 20:17:49.439043 13879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:49.439620 13885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18d4922401a54c36ba652e1f0330a41f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d4922401a54c36ba652e1f0330a41f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 45275 } } }
I20260620 20:17:49.439751 13885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:49.439854 13886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18d4922401a54c36ba652e1f0330a41f. Latest consensus state: current_term: 1 leader_uuid: "18d4922401a54c36ba652e1f0330a41f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d4922401a54c36ba652e1f0330a41f" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 45275 } } }
I20260620 20:17:49.440163 13886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:49.440456 13892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:49.441321 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:49.444373 13892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:49.450203 13892 catalog_manager.cc:1347] Generated new cluster ID: a1d0533b4c2a43e8b02ab651b71a60bb
I20260620 20:17:49.450299 13892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:49.451617 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:49.443539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53783 (local address 127.24.20.190:45275)
0620 20:17:49.443641 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:49.443646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:49.443856 (+ 210us) server_negotiation.cc:407] Connection header received
0620 20:17:49.443977 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:49.443980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:49.444019 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:49.444116 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:49.444930 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:49.445478 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:49.446303 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:49.446635 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:49.446750 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:49.448126 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:49.448130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:49.448133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:49.448156 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:49.449640 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:49.450217 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:49.450224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:49.450226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:49.450309 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:49.450698 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:49.450701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:49.450703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:49.450912 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:49.451060 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:49.451360 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 20:17:49.451459 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:17:49.468854 13892 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:49.469748 13892 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:49.477313 13892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18d4922401a54c36ba652e1f0330a41f: Generated new TSK 0
I20260620 20:17:49.478082 13892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:49.487948 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 13811
W20260620 20:17:49.492558 13796 connection.cc:535] client connection to 127.24.20.190:45275 recv error: Network error: failed to read from TLS socket (remote: 127.24.20.190:45275): Connection reset by peer (error 104)
2026-06-20T20:17:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:49 dist-test-slave-pflr krb5kdc[13910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:49 dist-test-slave-pflr krb5kdc[13910](info): set up 2 sockets
Jun 20 20:17:49 dist-test-slave-pflr krb5kdc[13910](info): commencing operation
krb5kdc: starting...
W20260620 20:17:50.529157 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:50 dist-test-slave-pflr krb5kdc[13910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:17:50.651188 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:33701
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:33701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:50.762146 13923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:50.762446 13923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:50.762488 13923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:50.766615 13923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:50.766669 13923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:50.766690 13923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:50.766705 13923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:50.766755 13923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:50.771981 13923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:33701
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:33701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:50.773048 13923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:50.774046 13923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:50.782574 13928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:50.782857 13929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:50.782881 13931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:50.783123 13923 server_base.cc:1034] running on GCE node
I20260620 20:17:50.783653 13923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:50.784866 13923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:50.786049 13923 hybrid_clock.cc:648] HybridClock initialized: now 1781986670786021 us; error 40 us; skew 500 ppm
Jun 20 20:17:50 dist-test-slave-pflr krb5kdc[13910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986670, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:50.788956 13923 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:17:50.789837 13923 webserver.cc:460] Webserver started at http://127.24.20.190:36185/ using document root <none> and password file <none>
I20260620 20:17:50.790375 13923 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:50.790441 13923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:50.790669 13923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:50.792718 13923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "cc9fca68b33a406fae64fbc7aeb9f8d2"
format_stamp: "Formatted at 2026-06-20 20:17:50 on dist-test-slave-pflr"
server_key: "4f3aa87d5de59b4128c3adeb161e897f"
server_key_iv: "58bf2df773e59862858b973cda1e29f8"
server_key_version: "encryptionkey@0"
I20260620 20:17:50.793269 13923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "cc9fca68b33a406fae64fbc7aeb9f8d2"
format_stamp: "Formatted at 2026-06-20 20:17:50 on dist-test-slave-pflr"
server_key: "4f3aa87d5de59b4128c3adeb161e897f"
server_key_iv: "58bf2df773e59862858b973cda1e29f8"
server_key_version: "encryptionkey@0"
I20260620 20:17:50.796888 13923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:17:50.799211 13938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:50.800448 13923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:50.800663 13923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "cc9fca68b33a406fae64fbc7aeb9f8d2"
format_stamp: "Formatted at 2026-06-20 20:17:50 on dist-test-slave-pflr"
server_key: "4f3aa87d5de59b4128c3adeb161e897f"
server_key_iv: "58bf2df773e59862858b973cda1e29f8"
server_key_version: "encryptionkey@0"
I20260620 20:17:50.800776 13923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:50.831207 13923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:50.834620 13923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:50.834833 13923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:50.841987 13923 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:33701
I20260620 20:17:50.842003 13990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:33701 every 8 connection(s)
I20260620 20:17:50.842945 13923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:50.845814 13991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:50.847548 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 13923
I20260620 20:17:50.847702 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:50.847982 24658 external_mini_cluster.cc:1428] Setting key 6510825777cfb16b02e987c13c34a355
I20260620 20:17:50.852250 13991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2: Bootstrap starting.
I20260620 20:17:50.854982 13991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:50.855708 13991 log.cc:824] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:50.856926 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:50.849354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41445 (local address 127.24.20.190:33701)
0620 20:17:50.849761 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:50.849777 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:50.849811 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:17:50.850436 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:50.850453 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:50.850798 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:50.851070 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:50.851485 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:50.852514 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:50.853752 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:50.854195 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:50.854309 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:50.855198 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:50.855203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:50.855214 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:50.855584 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:50.855635 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:50.855656 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:50.855759 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:50.856005 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 20:17:50.856147 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":130,"threads_started":1}
I20260620 20:17:50.857823 13991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2: No bootstrap required, opened a new log
I20260620 20:17:50.859704 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:50.861590 13991 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9fca68b33a406fae64fbc7aeb9f8d2" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33701 } }
I20260620 20:17:50.861898 13991 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:50.861972 13991 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9fca68b33a406fae64fbc7aeb9f8d2, State: Initialized, Role: FOLLOWER
I20260620 20:17:50.862480 13991 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9fca68b33a406fae64fbc7aeb9f8d2" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33701 } }
I20260620 20:17:50.862604 13991 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:50.862804 13991 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:50.863080 13991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:50.864053 13991 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9fca68b33a406fae64fbc7aeb9f8d2" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33701 } }
I20260620 20:17:50.864387 13991 leader_election.cc:302] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc9fca68b33a406fae64fbc7aeb9f8d2; no voters:
I20260620 20:17:50.864776 13991 leader_election.cc:288] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:50.864845 14001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:50.865037 14001 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [term 1 LEADER]: Becoming Leader. State: Replica: cc9fca68b33a406fae64fbc7aeb9f8d2, State: Running, Role: LEADER
I20260620 20:17:50.865373 14001 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9fca68b33a406fae64fbc7aeb9f8d2" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33701 } }
I20260620 20:17:50.865952 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:50.861753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42153 (local address 127.24.20.190:33701)
0620 20:17:50.861860 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:50.861867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:50.862019 (+ 152us) server_negotiation.cc:407] Connection header received
0620 20:17:50.862131 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:50.862133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:50.862167 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:50.862256 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:50.862529 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:50.863218 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:50.864036 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:50.864358 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:50.864470 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:50.865371 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:50.865374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:50.865377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:50.865389 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:50.865417 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:50.865429 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:50.865505 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:50.865811 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 20:17:50.865846 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:17:50.866230 13991 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:50.867408 14003 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc9fca68b33a406fae64fbc7aeb9f8d2. Latest consensus state: current_term: 1 leader_uuid: "cc9fca68b33a406fae64fbc7aeb9f8d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9fca68b33a406fae64fbc7aeb9f8d2" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33701 } } }
I20260620 20:17:50.867535 14003 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:50.868238 14010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:50.868274 14002 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc9fca68b33a406fae64fbc7aeb9f8d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc9fca68b33a406fae64fbc7aeb9f8d2" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 33701 } } }
I20260620 20:17:50.868360 14002 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:50.870841 14010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:50.875662 14010 catalog_manager.cc:1347] Generated new cluster ID: afe53c62878044a4aa825c8ee8dbba46
I20260620 20:17:50.875723 14010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:50.889864 14010 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:50.890894 14010 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:50.894404 14010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc9fca68b33a406fae64fbc7aeb9f8d2: Generated new TSK 0
I20260620 20:17:50.895020 14010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:50.902706 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 13923
W20260620 20:17:50.907277 13908 connection.cc:535] client connection to 127.24.20.190:33701 recv error: Network error: failed to read from TLS socket (remote: 127.24.20.190:33701): Connection reset by peer (error 104)
2026-06-20T20:17:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:50 dist-test-slave-pflr krb5kdc[14022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:50 dist-test-slave-pflr krb5kdc[14022](info): set up 2 sockets
Jun 20 20:17:50 dist-test-slave-pflr krb5kdc[14022](info): commencing operation
krb5kdc: starting...
W20260620 20:17:51.953091 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:52 dist-test-slave-pflr krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:17:52.072266 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:34601
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:34601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:52.180030 14035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:52.180337 14035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:52.180423 14035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:52.184437 14035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:52.184536 14035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:52.184576 14035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:52.184629 14035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:52.184666 14035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:52.190017 14035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:34601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:34601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:52.191071 14035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:52.192147 14035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:52.200452 14040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:52.200474 14041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:52.200521 14035 server_base.cc:1034] running on GCE node
W20260620 20:17:52.200734 14043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:52.201237 14035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:52.202340 14035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:52.203567 14035 hybrid_clock.cc:648] HybridClock initialized: now 1781986672203550 us; error 30 us; skew 500 ppm
Jun 20 20:17:52 dist-test-slave-pflr krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986672, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:52.206517 14035 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:17:52.207312 14035 webserver.cc:460] Webserver started at http://127.24.20.190:33289/ using document root <none> and password file <none>
I20260620 20:17:52.207870 14035 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:52.207929 14035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:52.208091 14035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:52.210047 14035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "a052f4efc81849f8a2e0f15ea8c1f66b"
format_stamp: "Formatted at 2026-06-20 20:17:52 on dist-test-slave-pflr"
server_key: "3fd41d087dfa80e4ac423cc2bbaf5f09"
server_key_iv: "bef480aad46f759eddaee8937720e9ea"
server_key_version: "encryptionkey@0"
I20260620 20:17:52.210538 14035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "a052f4efc81849f8a2e0f15ea8c1f66b"
format_stamp: "Formatted at 2026-06-20 20:17:52 on dist-test-slave-pflr"
server_key: "3fd41d087dfa80e4ac423cc2bbaf5f09"
server_key_iv: "bef480aad46f759eddaee8937720e9ea"
server_key_version: "encryptionkey@0"
I20260620 20:17:52.214002 14035 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:17:52.216202 14050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:52.217451 14035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:17:52.217592 14035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "a052f4efc81849f8a2e0f15ea8c1f66b"
format_stamp: "Formatted at 2026-06-20 20:17:52 on dist-test-slave-pflr"
server_key: "3fd41d087dfa80e4ac423cc2bbaf5f09"
server_key_iv: "bef480aad46f759eddaee8937720e9ea"
server_key_version: "encryptionkey@0"
I20260620 20:17:52.217715 14035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:52.222164 14035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:52.224962 14035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:52.225108 14035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:52.231838 14035 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:34601
I20260620 20:17:52.231848 14102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:34601 every 8 connection(s)
I20260620 20:17:52.232849 14035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:52.235461 14103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:52.238204 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 14035
I20260620 20:17:52.238346 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:52.238641 24658 external_mini_cluster.cc:1428] Setting key 15fe372257d0aace866816e891857523
I20260620 20:17:52.240912 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b: Bootstrap starting.
I20260620 20:17:52.243404 14103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:52.243755 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:52.239977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36727 (local address 127.24.20.190:34601)
0620 20:17:52.240372 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:52.240387 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:52.240421 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:17:52.241125 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:52.241144 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:52.241409 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:52.241676 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:52.242528 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:52.242532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:52.242544 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:52.242553 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:52.242603 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:52.242621 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:52.242727 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:52.242857 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 20:17:52.243045 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":132,"threads_started":1}
I20260620 20:17:52.244280 14103 log.cc:824] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:52.246243 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b: No bootstrap required, opened a new log
I20260620 20:17:52.246515 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:52.250203 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:52.248558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46549 (local address 127.24.20.190:34601)
0620 20:17:52.248701 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:52.248705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:52.248923 (+ 218us) server_negotiation.cc:407] Connection header received
0620 20:17:52.249026 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:52.249031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:52.249073 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:52.249171 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:52.249862 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:52.249866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:52.249869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:52.249880 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:52.249904 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:52.249917 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:52.250007 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:52.250069 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 20:17:52.250101 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:17:52.250361 14103 raft_consensus.cc:348] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a052f4efc81849f8a2e0f15ea8c1f66b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34601 } }
I20260620 20:17:52.250615 14103 raft_consensus.cc:374] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:52.250708 14103 raft_consensus.cc:729] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a052f4efc81849f8a2e0f15ea8c1f66b, State: Initialized, Role: FOLLOWER
I20260620 20:17:52.251101 14103 consensus_queue.cc:260] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a052f4efc81849f8a2e0f15ea8c1f66b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34601 } }
I20260620 20:17:52.251257 14103 raft_consensus.cc:388] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:52.251351 14103 raft_consensus.cc:482] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:52.251446 14103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:52.252295 14103 raft_consensus.cc:504] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a052f4efc81849f8a2e0f15ea8c1f66b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34601 } }
I20260620 20:17:52.252686 14103 leader_election.cc:302] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a052f4efc81849f8a2e0f15ea8c1f66b; no voters:
I20260620 20:17:52.252995 14103 leader_election.cc:288] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:52.253085 14113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:52.253286 14113 raft_consensus.cc:686] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [term 1 LEADER]: Becoming Leader. State: Replica: a052f4efc81849f8a2e0f15ea8c1f66b, State: Running, Role: LEADER
I20260620 20:17:52.253624 14113 consensus_queue.cc:237] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a052f4efc81849f8a2e0f15ea8c1f66b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34601 } }
I20260620 20:17:52.254163 14103 sys_catalog.cc:564] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:52.255218 14115 sys_catalog.cc:455] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a052f4efc81849f8a2e0f15ea8c1f66b. Latest consensus state: current_term: 1 leader_uuid: "a052f4efc81849f8a2e0f15ea8c1f66b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a052f4efc81849f8a2e0f15ea8c1f66b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34601 } } }
I20260620 20:17:52.255339 14115 sys_catalog.cc:458] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:52.255609 14114 sys_catalog.cc:455] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a052f4efc81849f8a2e0f15ea8c1f66b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a052f4efc81849f8a2e0f15ea8c1f66b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34601 } } }
I20260620 20:17:52.255705 14122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:52.255767 14114 sys_catalog.cc:458] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:52.258445 14122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:52.263375 14122 catalog_manager.cc:1347] Generated new cluster ID: d2d5ac4585f54cfb99dc4307ca53da6b
I20260620 20:17:52.263447 14122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:52.279191 14122 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:52.279461 14122 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:52.296123 14122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a052f4efc81849f8a2e0f15ea8c1f66b: Generated new TSK 0
I20260620 20:17:52.296876 14122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:52.307561 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 14035
2026-06-20T20:17:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:52 dist-test-slave-pflr krb5kdc[14134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:52 dist-test-slave-pflr krb5kdc[14134](info): set up 2 sockets
Jun 20 20:17:52 dist-test-slave-pflr krb5kdc[14134](info): commencing operation
krb5kdc: starting...
W20260620 20:17:53.337736 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:53 dist-test-slave-pflr krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:17:53.461755 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:42067
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:53.575155 14147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:53.575412 14147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:53.575453 14147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:53.579479 14147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:53.579540 14147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:53.579559 14147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:53.579605 14147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:53.579638 14147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:53.584856 14147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:42067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:53.585808 14147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:53.586804 14147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:53.595213 14152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:53.595239 14153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:53.595513 14155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:53.595335 14147 server_base.cc:1034] running on GCE node
I20260620 20:17:53.596019 14147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:53.597240 14147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:53.598441 14147 hybrid_clock.cc:648] HybridClock initialized: now 1781986673598428 us; error 26 us; skew 500 ppm
Jun 20 20:17:53 dist-test-slave-pflr krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986673, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:53.610960 14147 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:17:53.611831 14147 webserver.cc:460] Webserver started at http://127.24.20.190:39363/ using document root <none> and password file <none>
I20260620 20:17:53.612380 14147 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:53.612475 14147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:53.612766 14147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:53.614748 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "6d019f9617b445b1b640ff6212f15f0c"
format_stamp: "Formatted at 2026-06-20 20:17:53 on dist-test-slave-pflr"
server_key: "d87656a23bb29a301be0e8f363045069"
server_key_iv: "7803d9bd222393022d3faf76ae75a787"
server_key_version: "encryptionkey@0"
I20260620 20:17:53.615311 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "6d019f9617b445b1b640ff6212f15f0c"
format_stamp: "Formatted at 2026-06-20 20:17:53 on dist-test-slave-pflr"
server_key: "d87656a23bb29a301be0e8f363045069"
server_key_iv: "7803d9bd222393022d3faf76ae75a787"
server_key_version: "encryptionkey@0"
I20260620 20:17:53.619040 14147 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:17:53.621325 14162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:53.622504 14147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 20:17:53.622653 14147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "6d019f9617b445b1b640ff6212f15f0c"
format_stamp: "Formatted at 2026-06-20 20:17:53 on dist-test-slave-pflr"
server_key: "d87656a23bb29a301be0e8f363045069"
server_key_iv: "7803d9bd222393022d3faf76ae75a787"
server_key_version: "encryptionkey@0"
I20260620 20:17:53.622763 14147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:53.644248 14147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:53.647269 14147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:53.647471 14147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:53.655589 14147 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:42067
I20260620 20:17:53.655611 14214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:42067 every 8 connection(s)
I20260620 20:17:53.656646 14147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:53.658043 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 14147
I20260620 20:17:53.658207 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:53.658519 24658 external_mini_cluster.cc:1428] Setting key f25c7c881198b01a31cac2d9492e7a43
I20260620 20:17:53.659754 14215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:53.665958 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c: Bootstrap starting.
Jun 20 20:17:53 dist-test-slave-pflr krb5kdc[14134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:17:53.668694 14215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:53.669534 14215 log.cc:824] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:53.671307 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c: No bootstrap required, opened a new log
I20260620 20:17:53.674894 14215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d019f9617b445b1b640ff6212f15f0c" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42067 } }
I20260620 20:17:53.675140 14215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:53.675235 14215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d019f9617b445b1b640ff6212f15f0c, State: Initialized, Role: FOLLOWER
I20260620 20:17:53.675637 14215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d019f9617b445b1b640ff6212f15f0c" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42067 } }
I20260620 20:17:53.675786 14215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:53.675871 14215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:53.675987 14215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:53.676883 14215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d019f9617b445b1b640ff6212f15f0c" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42067 } }
I20260620 20:17:53.677233 14215 leader_election.cc:302] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d019f9617b445b1b640ff6212f15f0c; no voters:
I20260620 20:17:53.677597 14215 leader_election.cc:288] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:53.677667 14220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:53.677858 14220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [term 1 LEADER]: Becoming Leader. State: Replica: 6d019f9617b445b1b640ff6212f15f0c, State: Running, Role: LEADER
I20260620 20:17:53.678196 14220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d019f9617b445b1b640ff6212f15f0c" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42067 } }
I20260620 20:17:53.678514 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:53.659910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53377 (local address 127.24.20.190:42067)
0620 20:17:53.660327 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:53.660360 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:53.660398 (+ 38us) server_negotiation.cc:407] Connection header received
0620 20:17:53.661141 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:53.661160 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:53.661521 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:53.661815 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:53.662772 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:53.663924 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:53.665179 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:53.665723 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:53.665864 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:53.668713 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:53.668719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:53.668731 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:53.668757 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:53.675125 (+ 6368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:53.675705 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:53.675715 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:53.675720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:53.675897 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:53.676209 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:53.676212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:53.676214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:53.676586 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:53.676844 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:53.677112 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 20:17:53.677432 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":149,"threads_started":1}
I20260620 20:17:53.678805 14215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:53.679947 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d019f9617b445b1b640ff6212f15f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d019f9617b445b1b640ff6212f15f0c" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42067 } } }
I20260620 20:17:53.679956 14222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d019f9617b445b1b640ff6212f15f0c. Latest consensus state: current_term: 1 leader_uuid: "6d019f9617b445b1b640ff6212f15f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d019f9617b445b1b640ff6212f15f0c" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 42067 } } }
I20260620 20:17:53.680086 14222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:53.680087 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:53.680480 14229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:53.682138 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:53.683756 14229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:53.689344 14229 catalog_manager.cc:1347] Generated new cluster ID: dd23a44cfafa476f9ffce80a948c203b
I20260620 20:17:53.689448 14229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:53.693472 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:53.684348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35899 (local address 127.24.20.190:42067)
0620 20:17:53.684486 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:53.684490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:53.684884 (+ 394us) server_negotiation.cc:407] Connection header received
0620 20:17:53.684989 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:53.684992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:53.685040 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:53.685133 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:53.685930 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:53.686694 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:53.687943 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:53.688264 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:53.688367 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:53.689946 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:53.689951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:53.689954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:53.689979 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:53.691588 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:53.692082 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:53.692085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:53.692087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:53.692166 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:53.692485 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:53.692488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:53.692490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:53.692736 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:53.692839 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:53.693197 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 20:17:53.693311 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:17:53.721892 14229 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:53.722816 14229 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:53.727978 14229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d019f9617b445b1b640ff6212f15f0c: Generated new TSK 0
I20260620 20:17:53.728641 14229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:53.742142 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 14147
2026-06-20T20:17:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:53 dist-test-slave-pflr krb5kdc[14246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:53 dist-test-slave-pflr krb5kdc[14246](info): set up 2 sockets
Jun 20 20:17:53 dist-test-slave-pflr krb5kdc[14246](info): commencing operation
krb5kdc: starting...
W20260620 20:17:54.789229 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:54 dist-test-slave-pflr krb5kdc[14246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:17:54.912822 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:34737
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:34737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:55.019517 14259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:55.019826 14259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:55.019902 14259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:55.023945 14259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:55.024049 14259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:55.024088 14259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:55.024121 14259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:55.024149 14259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:55.029232 14259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:34737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:34737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:55.030241 14259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:55.031240 14259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:55.039609 14267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:55.039661 14264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:55.039781 14259 server_base.cc:1034] running on GCE node
W20260620 20:17:55.039947 14265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:55.040468 14259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:55.041551 14259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:55.042778 14259 hybrid_clock.cc:648] HybridClock initialized: now 1781986675042764 us; error 24 us; skew 500 ppm
Jun 20 20:17:55 dist-test-slave-pflr krb5kdc[14246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986675, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:55.045570 14259 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:17:55.046428 14259 webserver.cc:460] Webserver started at http://127.24.20.190:33223/ using document root <none> and password file <none>
I20260620 20:17:55.046981 14259 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:55.047075 14259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:55.047300 14259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:55.049348 14259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "d24b17e00d504fd5b7a520e5bab3fa62"
format_stamp: "Formatted at 2026-06-20 20:17:55 on dist-test-slave-pflr"
server_key: "4ce1a5ce379f56789d58e5e6a100e461"
server_key_iv: "9e34169fb87f272093affbeed49f6381"
server_key_version: "encryptionkey@0"
I20260620 20:17:55.049926 14259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "d24b17e00d504fd5b7a520e5bab3fa62"
format_stamp: "Formatted at 2026-06-20 20:17:55 on dist-test-slave-pflr"
server_key: "4ce1a5ce379f56789d58e5e6a100e461"
server_key_iv: "9e34169fb87f272093affbeed49f6381"
server_key_version: "encryptionkey@0"
I20260620 20:17:55.053959 14259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 20:17:55.056394 14274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:55.057713 14259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:17:55.057868 14259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "d24b17e00d504fd5b7a520e5bab3fa62"
format_stamp: "Formatted at 2026-06-20 20:17:55 on dist-test-slave-pflr"
server_key: "4ce1a5ce379f56789d58e5e6a100e461"
server_key_iv: "9e34169fb87f272093affbeed49f6381"
server_key_version: "encryptionkey@0"
I20260620 20:17:55.058000 14259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:55.090622 14259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:55.093796 14259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:55.094022 14259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:55.101265 14259 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:34737
I20260620 20:17:55.101296 14326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:34737 every 8 connection(s)
I20260620 20:17:55.102193 14259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:55.104997 14327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:55.108978 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 14259
I20260620 20:17:55.109155 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:55.109418 24658 external_mini_cluster.cc:1428] Setting key 66cb8fe41db57c52b772cfcc8b2ace4b
I20260620 20:17:55.110718 14327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62: Bootstrap starting.
I20260620 20:17:55.113399 14327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:55.114274 14327 log.cc:824] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:55.116223 14327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62: No bootstrap required, opened a new log
I20260620 20:17:55.119753 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:55.110893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33227 (local address 127.24.20.190:34737)
0620 20:17:55.111297 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:55.111315 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:55.111356 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:17:55.112185 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:55.112209 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:55.112647 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:55.112998 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:55.113409 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:55.114843 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:55.115792 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:55.116448 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:55.116659 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:55.117730 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:55.117738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:55.117756 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:55.118229 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:55.118294 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:55.118322 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:55.118422 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:55.118570 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 20:17:55.118765 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
I20260620 20:17:55.120666 14327 raft_consensus.cc:348] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24b17e00d504fd5b7a520e5bab3fa62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34737 } }
I20260620 20:17:55.120903 14327 raft_consensus.cc:374] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:55.120999 14327 raft_consensus.cc:729] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24b17e00d504fd5b7a520e5bab3fa62, State: Initialized, Role: FOLLOWER
I20260620 20:17:55.121452 14327 consensus_queue.cc:260] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24b17e00d504fd5b7a520e5bab3fa62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34737 } }
I20260620 20:17:55.121636 14327 raft_consensus.cc:388] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:55.121760 14327 raft_consensus.cc:482] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:55.121866 14327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:55.122640 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:55.122866 14327 raft_consensus.cc:504] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24b17e00d504fd5b7a520e5bab3fa62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34737 } }
I20260620 20:17:55.123205 14327 leader_election.cc:302] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d24b17e00d504fd5b7a520e5bab3fa62; no voters:
I20260620 20:17:55.123522 14327 leader_election.cc:288] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:55.123585 14332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:55.123757 14332 raft_consensus.cc:686] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [term 1 LEADER]: Becoming Leader. State: Replica: d24b17e00d504fd5b7a520e5bab3fa62, State: Running, Role: LEADER
I20260620 20:17:55.124092 14332 consensus_queue.cc:237] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24b17e00d504fd5b7a520e5bab3fa62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34737 } }
I20260620 20:17:55.124732 14327 sys_catalog.cc:564] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:55.125737 14333 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d24b17e00d504fd5b7a520e5bab3fa62. Latest consensus state: current_term: 1 leader_uuid: "d24b17e00d504fd5b7a520e5bab3fa62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24b17e00d504fd5b7a520e5bab3fa62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34737 } } }
I20260620 20:17:55.125875 14333 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:55.126096 14334 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d24b17e00d504fd5b7a520e5bab3fa62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24b17e00d504fd5b7a520e5bab3fa62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34737 } } }
I20260620 20:17:55.126206 14334 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:55.126447 14346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:55.129511 14346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:55.134202 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:55.125225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59321 (local address 127.24.20.190:34737)
0620 20:17:55.125322 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:55.125326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:55.129889 (+ 4563us) server_negotiation.cc:407] Connection header received
0620 20:17:55.130092 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:55.130095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:55.130144 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:55.130195 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:55.130596 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:55.131345 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:55.132212 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:55.132453 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:55.132570 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:55.133750 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:55.133754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:55.133756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:55.133768 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:55.133794 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:55.133811 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:55.133895 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:55.134017 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 20:17:55.134066 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:17:55.135723 14346 catalog_manager.cc:1347] Generated new cluster ID: 56e54e14b5ad46edada5553771ae8833
I20260620 20:17:55.135788 14346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:55.168995 14346 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:55.170269 14346 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:55.175541 14346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d24b17e00d504fd5b7a520e5bab3fa62: Generated new TSK 0
I20260620 20:17:55.176352 14346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:55.187106 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 14259
2026-06-20T20:17:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:55 dist-test-slave-pflr krb5kdc[14358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:55 dist-test-slave-pflr krb5kdc[14358](info): set up 2 sockets
Jun 20 20:17:55 dist-test-slave-pflr krb5kdc[14358](info): commencing operation
krb5kdc: starting...
W20260620 20:17:56.221319 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:56 dist-test-slave-pflr krb5kdc[14358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:17:56.339699 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:40687
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:40687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:56.447417 14371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:56.447692 14371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:56.447764 14371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:56.451735 14371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:56.451822 14371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:56.451865 14371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:56.451929 14371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:56.451962 14371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:56.457813 14371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:40687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:40687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:56.458829 14371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:56.459834 14371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:56.468729 14377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:56.468817 14376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:56.468854 14379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:56.469024 14371 server_base.cc:1034] running on GCE node
I20260620 20:17:56.469540 14371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:56.470607 14371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:56.471776 14371 hybrid_clock.cc:648] HybridClock initialized: now 1781986676471766 us; error 27 us; skew 500 ppm
Jun 20 20:17:56 dist-test-slave-pflr krb5kdc[14358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986676, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:56.474515 14371 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:17:56.475376 14371 webserver.cc:460] Webserver started at http://127.24.20.190:42337/ using document root <none> and password file <none>
I20260620 20:17:56.475871 14371 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:56.475930 14371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:56.476080 14371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:56.478086 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "c4ebf8a50bb14f61869eadad48782ee0"
format_stamp: "Formatted at 2026-06-20 20:17:56 on dist-test-slave-pflr"
server_key: "c27ac3172e2a5260902882e433e41c6d"
server_key_iv: "decfa78f0b296544dde316388fa2216f"
server_key_version: "encryptionkey@0"
I20260620 20:17:56.478588 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "c4ebf8a50bb14f61869eadad48782ee0"
format_stamp: "Formatted at 2026-06-20 20:17:56 on dist-test-slave-pflr"
server_key: "c27ac3172e2a5260902882e433e41c6d"
server_key_iv: "decfa78f0b296544dde316388fa2216f"
server_key_version: "encryptionkey@0"
I20260620 20:17:56.482120 14371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 20:17:56.484372 14386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:56.485646 14371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:17:56.485786 14371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "c4ebf8a50bb14f61869eadad48782ee0"
format_stamp: "Formatted at 2026-06-20 20:17:56 on dist-test-slave-pflr"
server_key: "c27ac3172e2a5260902882e433e41c6d"
server_key_iv: "decfa78f0b296544dde316388fa2216f"
server_key_version: "encryptionkey@0"
I20260620 20:17:56.485894 14371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:56.490305 14371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:56.493216 14371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:56.493458 14371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:56.500496 14371 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:40687
I20260620 20:17:56.500516 14438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:40687 every 8 connection(s)
I20260620 20:17:56.501420 14371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:56.504091 14439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:56.504966 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 14371
I20260620 20:17:56.505108 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:56.505415 24658 external_mini_cluster.cc:1428] Setting key e850e93d0400784aba02a8ce19ce3647
I20260620 20:17:56.510433 14439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0: Bootstrap starting.
I20260620 20:17:56.510643 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:56.506831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48437 (local address 127.24.20.190:40687)
0620 20:17:56.507272 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:56.507286 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:56.507320 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:17:56.507953 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:56.507969 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:56.508239 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:56.508585 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:56.509438 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:56.509442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:56.509453 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:56.509464 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:56.509526 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:56.509545 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:56.509695 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:56.509775 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 20:17:56.509925 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":129,"threads_started":1}
I20260620 20:17:56.512943 14439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:56.513633 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:56.513759 14439 log.cc:824] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:56.515656 14439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0: No bootstrap required, opened a new log
I20260620 20:17:56.517421 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:56.515733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33085 (local address 127.24.20.190:40687)
0620 20:17:56.515874 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:56.515879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:56.516064 (+ 185us) server_negotiation.cc:407] Connection header received
0620 20:17:56.516182 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:56.516186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:56.516222 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:56.516307 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:56.517028 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:56.517031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:56.517034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:17:56.517045 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:56.517074 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:17:56.517085 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:56.517179 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:56.517251 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 20:17:56.517288 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:17:56.519564 14439 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebf8a50bb14f61869eadad48782ee0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40687 } }
I20260620 20:17:56.519764 14439 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:17:56.519845 14439 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ebf8a50bb14f61869eadad48782ee0, State: Initialized, Role: FOLLOWER
I20260620 20:17:56.520262 14439 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebf8a50bb14f61869eadad48782ee0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40687 } }
I20260620 20:17:56.520377 14439 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:17:56.520467 14439 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:17:56.520581 14439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:56.521659 14439 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebf8a50bb14f61869eadad48782ee0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40687 } }
I20260620 20:17:56.522001 14439 leader_election.cc:302] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4ebf8a50bb14f61869eadad48782ee0; no voters:
I20260620 20:17:56.522290 14439 leader_election.cc:288] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:17:56.522420 14449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:56.522693 14449 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [term 1 LEADER]: Becoming Leader. State: Replica: c4ebf8a50bb14f61869eadad48782ee0, State: Running, Role: LEADER
I20260620 20:17:56.523058 14449 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebf8a50bb14f61869eadad48782ee0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40687 } }
I20260620 20:17:56.523538 14439 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:56.524742 14450 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4ebf8a50bb14f61869eadad48782ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebf8a50bb14f61869eadad48782ee0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40687 } } }
I20260620 20:17:56.524863 14450 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:56.525161 14451 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ebf8a50bb14f61869eadad48782ee0. Latest consensus state: current_term: 1 leader_uuid: "c4ebf8a50bb14f61869eadad48782ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ebf8a50bb14f61869eadad48782ee0" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 40687 } } }
I20260620 20:17:56.525308 14451 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:56.525228 14458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:56.528092 14458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:56.533543 14458 catalog_manager.cc:1347] Generated new cluster ID: 1eb5dfb37cf2486f9d122f1f3b417a96
I20260620 20:17:56.533609 14458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:56.548830 14458 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:56.549175 14458 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:56.558717 14458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4ebf8a50bb14f61869eadad48782ee0: Generated new TSK 0
I20260620 20:17:56.559544 14458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:17:56.573349 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 14371
2026-06-20T20:17:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1387 ms)
[----------] 18 tests from AuthTokenIssuingTest (25375 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:56 dist-test-slave-pflr krb5kdc[14470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:56 dist-test-slave-pflr krb5kdc[14470](info): set up 2 sockets
Jun 20 20:17:56 dist-test-slave-pflr krb5kdc[14470](info): commencing operation
krb5kdc: starting...
W20260620 20:17:57.617288 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:17:57 dist-test-slave-pflr krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:17:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:17:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:17:57.735327 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:34681
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:34681,127.24.20.189:44769,127.24.20.188:39519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:17:57.844528 14483 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:17:57.844805 14483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:57.844827 14483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:57.844864 14483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:57.848734 14483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:57.848779 14483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:57.848799 14483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:57.848819 14483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:57.848834 14483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:57.853755 14483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:34681,127.24.20.189:44769,127.24.20.188:39519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:34681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:57.854645 14483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:57.855540 14483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:57.863801 14491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:57.863799 14488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:57.863830 14489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:57.864078 14483 server_base.cc:1034] running on GCE node
I20260620 20:17:57.864707 14483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:57.865814 14483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:57.867035 14483 hybrid_clock.cc:648] HybridClock initialized: now 1781986677867003 us; error 46 us; skew 500 ppm
Jun 20 20:17:57 dist-test-slave-pflr krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:57.869941 14483 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:17:57.870844 14483 webserver.cc:460] Webserver started at http://127.24.20.190:34677/ using document root <none> and password file <none>
I20260620 20:17:57.871533 14483 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:57.871639 14483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:57.871901 14483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:57.874215 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "8c2047ee3d464c8dbec38472352d0e62"
format_stamp: "Formatted at 2026-06-20 20:17:57 on dist-test-slave-pflr"
server_key: "a472dc95723244fc223dd945607a1908"
server_key_iv: "6dcda1d824043ce54f9a0024a02f261b"
server_key_version: "encryptionkey@0"
I20260620 20:17:57.874785 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "8c2047ee3d464c8dbec38472352d0e62"
format_stamp: "Formatted at 2026-06-20 20:17:57 on dist-test-slave-pflr"
server_key: "a472dc95723244fc223dd945607a1908"
server_key_iv: "6dcda1d824043ce54f9a0024a02f261b"
server_key_version: "encryptionkey@0"
I20260620 20:17:57.878500 14483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:17:57.880926 14498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:57.882185 14483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:17:57.882335 14483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "8c2047ee3d464c8dbec38472352d0e62"
format_stamp: "Formatted at 2026-06-20 20:17:57 on dist-test-slave-pflr"
server_key: "a472dc95723244fc223dd945607a1908"
server_key_iv: "6dcda1d824043ce54f9a0024a02f261b"
server_key_version: "encryptionkey@0"
I20260620 20:17:57.882507 14483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:57.898032 14483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:57.900910 14483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:57.901108 14483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:57.907902 14483 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:34681
I20260620 20:17:57.908191 14550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:34681 every 8 connection(s)
I20260620 20:17:57.908934 14483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:17:57.910995 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 14483
I20260620 20:17:57.911103 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:17:57.911418 24658 external_mini_cluster.cc:1428] Setting key 8e58f6bf58186ed60817f36f4a503322
I20260620 20:17:57.911775 14551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:57.915570 14551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.20.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.189@KRBTEST.COM" created.
Jun 20 20:17:57 dist-test-slave-pflr krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:17:57.931295 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:57.918051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:34681 (local address 127.0.0.1:37926)
0620 20:17:57.918629 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:57.918665 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:57.919589 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:57.920359 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:57.920365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:57.920797 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:57.921042 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:57.921055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:57.922553 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:57.922557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:57.923273 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:57.923282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:57.923919 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:57.923922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:57.924034 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:57.924695 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:57.924710 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:57.926343 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:57.928289 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:57.928299 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:57.928308 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:57.928529 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:57.928874 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:57.928880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:57.928882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:57.929000 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:57.929581 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:57.929587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:57.929733 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 20:17:57.929899 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 20:17:57.930318 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":129,"spinlock_wait_cycles":16128,"thread_start_us":145,"threads_started":1}
I20260620 20:17:57.931295 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:57.918051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.24.20.190:34681)
0620 20:17:57.918478 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:57.918507 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:57.918781 (+ 274us) server_negotiation.cc:407] Connection header received
0620 20:17:57.919900 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:57.919904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:57.920151 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:57.920363 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:57.921228 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:57.922411 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:57.923402 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:57.923766 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:57.923935 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:57.926492 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:57.926505 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:57.926519 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:57.926538 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:57.928158 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:57.928663 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:57.928668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:57.928670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:57.928772 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:57.929102 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:57.929105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:57.929107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:57.929444 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:57.929638 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:57.929969 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 20:17:57.930267 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":282,"thread_start_us":143,"threads_started":1}
I20260620 20:17:57.933784 14551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:17:57.934507 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:57.934258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:44769 (local address 127.0.0.1:60442)
0620 20:17:57.934373 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:57.934448 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:44769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 20:17:57.934736 14502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.20.189:44769: connect: Connection refused (error 111)
I20260620 20:17:57.935529 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:57.935323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:44769 (local address 127.0.0.1:60454)
0620 20:17:57.935425 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:57.935469 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:44769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 20:17:57.935689 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.189:44769: Network error: Client connection negotiation failed: client connection to 127.24.20.189:44769: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.189.keytab.
Entry for principal kudu/127.24.20.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.189.keytab.
I20260620 20:17:57.937951 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.189:44769
--webserver_interface=127.24.20.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.189.keytab
--principal=kudu/127.24.20.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:34681,127.24.20.189:44769,127.24.20.188:39519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:17:57.984932 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } attempt: 1
I20260620 20:17:57.985867 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:57.985529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:44769 (local address 127.0.0.1:60464)
0620 20:17:57.985685 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:57.985778 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:44769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 20:17:57.986579 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:57.986370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:44769 (local address 127.0.0.1:60476)
0620 20:17:57.986474 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:57.986520 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:44769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 20:17:57.986732 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.189:44769: Network error: Client connection negotiation failed: client connection to 127.24.20.189:44769: connect: Connection refused (error 111)
W20260620 20:17:58.045552 14558 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:17:58.045811 14558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:58.045835 14558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:58.045938 14558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:17:58.049909 14558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:58.049963 14558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:58.049983 14558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:58.050040 14558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:58.050073 14558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:58.054911 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } attempt: 2
I20260620 20:17:58.055730 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.055446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:44769 (local address 127.0.0.1:60492)
0620 20:17:58.055595 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.055656 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:44769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 20:17:58.056032 14558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:34681,127.24.20.189:44769,127.24.20.188:39519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.189.keytab
--principal=kudu/127.24.20.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.189:44769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.20.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:58.056466 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.056263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:44769 (local address 127.0.0.1:60508)
0620 20:17:58.056357 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.056400 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:44769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 20:17:58.056670 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.189:44769: Network error: Client connection negotiation failed: client connection to 127.24.20.189:44769: connect: Connection refused (error 111)
I20260620 20:17:58.057204 14558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:58.058104 14558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:58.066256 14563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:58.066298 14566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:58.066308 14564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:58.066497 14558 server_base.cc:1034] running on GCE node
I20260620 20:17:58.067037 14558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:58.068060 14558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:58.069219 14558 hybrid_clock.cc:648] HybridClock initialized: now 1781986678069189 us; error 41 us; skew 500 ppm
Jun 20 20:17:58 dist-test-slave-pflr krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986678, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:58.071848 14558 init.cc:377] Logged in from keytab as kudu/127.24.20.189@KRBTEST.COM (short username kudu)
I20260620 20:17:58.072775 14558 webserver.cc:460] Webserver started at http://127.24.20.189:35099/ using document root <none> and password file <none>
I20260620 20:17:58.073290 14558 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:58.073381 14558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:58.073613 14558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:58.075546 14558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/data/instance:
uuid: "aca400f9df3544f3ab1e4173e951c43a"
format_stamp: "Formatted at 2026-06-20 20:17:58 on dist-test-slave-pflr"
server_key: "d2b723fce60881a35b0518b6d7097005"
server_key_iv: "5233c3578b07330682b686946283a2c1"
server_key_version: "encryptionkey@0"
I20260620 20:17:58.076097 14558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/wal/instance:
uuid: "aca400f9df3544f3ab1e4173e951c43a"
format_stamp: "Formatted at 2026-06-20 20:17:58 on dist-test-slave-pflr"
server_key: "d2b723fce60881a35b0518b6d7097005"
server_key_iv: "5233c3578b07330682b686946283a2c1"
server_key_version: "encryptionkey@0"
I20260620 20:17:58.079676 14558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:17:58.081980 14573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:58.083194 14558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 20:17:58.083331 14558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/wal
uuid: "aca400f9df3544f3ab1e4173e951c43a"
format_stamp: "Formatted at 2026-06-20 20:17:58 on dist-test-slave-pflr"
server_key: "d2b723fce60881a35b0518b6d7097005"
server_key_iv: "5233c3578b07330682b686946283a2c1"
server_key_version: "encryptionkey@0"
I20260620 20:17:58.083473 14558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:58.102911 14558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:58.105734 14558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:58.105935 14558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:58.112859 14558 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.189:44769
I20260620 20:17:58.113152 14625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.189:44769 every 8 connection(s)
I20260620 20:17:58.113803 14558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/data/info.pb
I20260620 20:17:58.116475 14626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:58.120065 14626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:17:58.123653 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 14558
I20260620 20:17:58.123814 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-1/wal/instance
I20260620 20:17:58.124114 24658 external_mini_cluster.cc:1428] Setting key f89d09d6cc22ab89712f329cfd235a2f
Jun 20 20:17:58 dist-test-slave-pflr krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986678, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.189@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.20.188@KRBTEST.COM; defaulting to no policy
I20260620 20:17:58.133667 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.121874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.24.20.190:34681)
0620 20:17:58.122016 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.122020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.122704 (+ 684us) server_negotiation.cc:407] Connection header received
0620 20:17:58.123617 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.123620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.123680 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.123782 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.125075 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.125607 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.126725 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.126965 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.127090 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.129500 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.129504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.129507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.129536 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.131635 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.132138 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.132143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.132145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.132283 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.132692 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.132694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.132695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.132861 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.132937 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.133443 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.133548 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.188@KRBTEST.COM" created.
I20260620 20:17:58.134480 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.122131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:34681 (local address 127.0.0.1:37928)
0620 20:17:58.122554 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.122596 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:58.123373 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:58.123976 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:58.123988 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:58.124369 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:58.124930 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.124944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.125741 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.125745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.126597 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.126606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.127125 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.127129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.127297 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.127887 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:58.127902 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:58.129339 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:58.131760 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.131765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.131778 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.132029 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.132391 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.132395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.132397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.132536 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.132955 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:58.132960 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:58.133181 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 20:17:58.133327 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 20:17:58.133570 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260620 20:17:58.135872 14626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:17:58 dist-test-slave-pflr krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986678, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.189@KRBTEST.COM for kudu/127.24.20.189@KRBTEST.COM
I20260620 20:17:58.144901 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.136380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:44769 (local address 127.0.0.1:60510)
0620 20:17:58.136546 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.136562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:58.136706 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:58.137138 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:58.137140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:58.137305 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:58.137383 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.137387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.138398 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.138401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.138934 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.138941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.139498 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.139502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.139597 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.140091 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:58.140101 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:58.141309 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:58.143442 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.143444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.143446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.143663 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.143974 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.143976 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.143977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.144047 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.144496 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:58.144498 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:58.144570 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 20:17:58.144709 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 20:17:58.144799 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 20:17:58.145062 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.136513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.24.20.189:44769)
0620 20:17:58.136818 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.136827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.136843 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:17:58.136872 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.136875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.137019 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.137158 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.137497 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.138270 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.139048 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.139379 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.139503 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.141452 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.141462 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.141475 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.141510 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.143340 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.143780 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.143789 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.143794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.143881 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.144127 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.144129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.144130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.144403 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.144533 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.144772 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.144880 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":131,"threads_started":1}
I20260620 20:17:58.146750 14626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:17:58.147361 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.147158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53578)
0620 20:17:58.147256 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.147315 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 20:17:58.147567 14576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
I20260620 20:17:58.147835 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } attempt: 3
I20260620 20:17:58.148363 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.148158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53594)
0620 20:17:58.148257 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.148305 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 20:17:58.148536 14626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:39519: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.188.keytab.
Entry for principal kudu/127.24.20.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.188.keytab.
I20260620 20:17:58.151489 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.188:39519
--webserver_interface=127.24.20.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.188.keytab
--principal=kudu/127.24.20.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:34681,127.24.20.189:44769,127.24.20.188:39519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:17:58 dist-test-slave-pflr krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.189@KRBTEST.COM
I20260620 20:17:58.156297 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.148282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.24.20.189:44769)
0620 20:17:58.148358 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.148363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.148404 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:17:58.148531 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.148534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.148574 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.148718 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.149144 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.149711 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.150615 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.150838 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.150920 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.153195 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.153201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.153206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.153233 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.154705 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.155152 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.155157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.155159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.155233 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.155528 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.155532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.155534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.155697 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.155804 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.156064 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.156163 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 20:17:58.156299 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.148248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:44769 (local address 127.0.0.1:60524)
0620 20:17:58.148345 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.148358 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:58.148445 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:58.148737 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:58.148740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:58.148931 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:58.149027 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.149033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.149819 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.149823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.150484 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.150503 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.150953 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.150959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.151069 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.151541 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:58.151559 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:58.153058 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:58.154837 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.154840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.154842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.155045 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.155340 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.155342 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.155345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.155426 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.155795 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:58.155798 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:58.155898 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 20:17:58.156015 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 20:17:58.156151 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 20:17:58.157383 14551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:17:58.158149 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.157849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53604)
0620 20:17:58.157998 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.158063 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 20:17:58.158946 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.158633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53614)
0620 20:17:58.158798 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.158867 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 20:17:58.159076 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:39519: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
I20260620 20:17:58.190253 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } attempt: 1
I20260620 20:17:58.191139 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.190808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53616)
0620 20:17:58.190978 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.191050 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 20:17:58.191876 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.191662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53620)
0620 20:17:58.191765 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.191816 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 20:17:58.192041 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:39519: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
I20260620 20:17:58.197800 14626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } attempt: 1
I20260620 20:17:58.198546 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.198254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53630)
0620 20:17:58.198422 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.198478 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 20:17:58.199345 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.199120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53632)
0620 20:17:58.199239 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.199292 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 20:17:58.199474 14626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:39519: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
W20260620 20:17:58.262840 14633 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:17:58.263154 14633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:17:58.263228 14633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:17:58.263309 14633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 20:17:58.267235 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } attempt: 2
I20260620 20:17:58.267624 14626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } attempt: 2
W20260620 20:17:58.268003 14633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:17:58.268087 14633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:17:58.268134 14633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:17:58.268169 14633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:17:58.268191 14633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:17:58.268201 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.267836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53644)
0620 20:17:58.268018 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.268101 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 20:17:58.268257 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.268001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53650)
0620 20:17:58.268111 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.268170 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 20:17:58.268951 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.268702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53664)
0620 20:17:58.268818 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.268877 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63,"thread_start_us":124,"threads_started":1}
I20260620 20:17:58.269125 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.268871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53670)
0620 20:17:58.269004 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.269054 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 20:17:58.269133 14551 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:39519: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
W20260620 20:17:58.269266 14626 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:39519: Network error: Client connection negotiation failed: client connection to 127.24.20.188:39519: connect: Connection refused (error 111)
I20260620 20:17:58.273727 14633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:34681,127.24.20.189:44769,127.24.20.188:39519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.188.keytab
--principal=kudu/127.24.20.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.188:39519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.20.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:17:58.274698 14633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:17:58.275643 14633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:17:58.284006 14640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:58.284044 14642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:17:58.284003 14639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:17:58.284094 14633 server_base.cc:1034] running on GCE node
I20260620 20:17:58.284891 14633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:17:58.285930 14633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:17:58.287180 14633 hybrid_clock.cc:648] HybridClock initialized: now 1781986678287165 us; error 22 us; skew 500 ppm
Jun 20 20:17:58 dist-test-slave-pflr krb5kdc[14470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986678, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:17:58.290001 14633 init.cc:377] Logged in from keytab as kudu/127.24.20.188@KRBTEST.COM (short username kudu)
I20260620 20:17:58.290796 14633 webserver.cc:460] Webserver started at http://127.24.20.188:44903/ using document root <none> and password file <none>
I20260620 20:17:58.291334 14633 fs_manager.cc:359] Metadata directory not provided
I20260620 20:17:58.291425 14633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:17:58.291693 14633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:17:58.293722 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/data/instance:
uuid: "b0072d1b8f9140fea698673ec677788b"
format_stamp: "Formatted at 2026-06-20 20:17:58 on dist-test-slave-pflr"
server_key: "8807bd8b8995b889bc77bdf2c1c28a89"
server_key_iv: "18f4d27e87fb8a18102a5d094bb3e0cf"
server_key_version: "encryptionkey@0"
I20260620 20:17:58.294270 14633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/wal/instance:
uuid: "b0072d1b8f9140fea698673ec677788b"
format_stamp: "Formatted at 2026-06-20 20:17:58 on dist-test-slave-pflr"
server_key: "8807bd8b8995b889bc77bdf2c1c28a89"
server_key_iv: "18f4d27e87fb8a18102a5d094bb3e0cf"
server_key_version: "encryptionkey@0"
I20260620 20:17:58.298599 14633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:17:58.300940 14649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:17:58.302191 14633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:17:58.302419 14633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/wal
uuid: "b0072d1b8f9140fea698673ec677788b"
format_stamp: "Formatted at 2026-06-20 20:17:58 on dist-test-slave-pflr"
server_key: "8807bd8b8995b889bc77bdf2c1c28a89"
server_key_iv: "18f4d27e87fb8a18102a5d094bb3e0cf"
server_key_version: "encryptionkey@0"
I20260620 20:17:58.302549 14633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:17:58.312450 14633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:17:58.315708 14633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:17:58.315922 14633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:17:58.323053 14633 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.188:39519
I20260620 20:17:58.323340 14701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.188:39519 every 8 connection(s)
I20260620 20:17:58.324111 14633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/data/info.pb
I20260620 20:17:58.326831 14702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:17:58.328004 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 14633
I20260620 20:17:58.328107 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986405628821-24658-0/minicluster-data/master-2/wal/instance
I20260620 20:17:58.328436 24658 external_mini_cluster.cc:1428] Setting key a22d97a1a3bf92a3965d97d8ebe8a0a3
I20260620 20:17:58.331283 14702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:17:58 dist-test-slave-pflr krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.188@KRBTEST.COM
Jun 20 20:17:58 dist-test-slave-pflr krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986678, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.188@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:17:58.342808 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.333179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.24.20.190:34681)
0620 20:17:58.333326 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.333330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.333570 (+ 240us) server_negotiation.cc:407] Connection header received
0620 20:17:58.333697 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.333700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.333747 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.333830 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.334433 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.335200 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.336066 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.336284 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.336406 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.339263 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.339270 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.339275 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.339302 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.340960 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.341417 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.341420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.341422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.341499 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.341821 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.341825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.341827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.342014 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.342133 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.342513 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.342627 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:17:58.343578 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.333074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:34681 (local address 127.0.0.1:37932)
0620 20:17:58.333443 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.333487 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:58.333607 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:58.333887 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:58.333891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:58.334195 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:58.334309 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.334314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.335317 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.335319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.335921 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.335928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.336440 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.336444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.336581 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.337189 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:58.337204 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:58.339084 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:58.341065 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.341073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.341082 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.341296 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.341599 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.341602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.341604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.341717 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.342130 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:58.342135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:58.342269 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 20:17:58.342456 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 20:17:58.342982 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":214,"thread_start_us":122,"threads_started":1}
I20260620 20:17:58.343577 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.329937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.24.20.188:39519)
0620 20:17:58.330367 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.330385 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.330426 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:17:58.331238 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.331260 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.331658 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.332002 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.332824 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.334205 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.335390 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.335848 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.336035 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.338703 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.338712 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.338730 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.338752 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.340876 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.341325 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.341331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.341333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.341443 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.341754 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.341759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.341761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.342121 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.342382 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.342568 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.342752 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":109,"threads_started":1}
I20260620 20:17:58.344945 14702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:17:58 dist-test-slave-pflr krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986678, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.188@KRBTEST.COM for kudu/127.24.20.189@KRBTEST.COM
Jun 20 20:17:58 dist-test-slave-pflr krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:17:58.354102 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.345455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:44769 (local address 127.0.0.1:60532)
0620 20:17:58.345542 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.345565 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:58.345633 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:58.345947 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:58.345950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:58.346072 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:58.346202 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.346208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.347214 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.347217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.347935 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.347942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.348482 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.348484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.348551 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.349034 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:58.349052 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:58.350554 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:58.352500 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.352503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.352504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.352754 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.353048 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.353050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.353051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.353120 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.353559 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:58.353561 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:58.353633 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 20:17:58.353768 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 20:17:58.353899 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 20:17:58.354142 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.345643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.24.20.189:44769)
0620 20:17:58.345760 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.345766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.345782 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:17:58.345830 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.345832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.345870 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.345935 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.346345 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.347113 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.348043 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.348368 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.348487 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.350710 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.350713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.350716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.350740 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.352384 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.352856 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.352860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.352863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.352951 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.353252 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.353254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.353255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.353410 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.353529 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.353828 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.353984 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:17:58.354885 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.345706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.24.20.190:34681)
0620 20:17:58.345792 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.345796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.346162 (+ 366us) server_negotiation.cc:407] Connection header received
0620 20:17:58.346203 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.346206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.346251 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.346329 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.346997 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.347688 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.348679 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.349006 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.349127 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.351509 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.351518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.351520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.351539 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.353085 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.353497 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.353503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.353505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.353592 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.353899 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.353902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.353904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.354094 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.354306 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.354659 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.354772 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 20:17:58.355146 14702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:17:58 dist-test-slave-pflr krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986678, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.188@KRBTEST.COM for kudu/127.24.20.188@KRBTEST.COM
I20260620 20:17:58.360656 14626 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } attempt: 3
Jun 20 20:17:58 dist-test-slave-pflr krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.189@KRBTEST.COM
I20260620 20:17:58.363489 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.355776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53702 (local address 127.24.20.188:39519)
0620 20:17:58.355863 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.355867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.355876 (+ 9us) server_negotiation.cc:407] Connection header received
0620 20:17:58.355902 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.355904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.355963 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.356048 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.356478 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.357076 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.357928 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.358145 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.358216 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.360194 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.360198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.360201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.360224 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.361888 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.362333 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.362337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.362360 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.362443 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.362742 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.362748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.362750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.362910 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.363033 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.363290 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.363390 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 20:17:58.363828 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.355587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53702)
0620 20:17:58.355688 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.355703 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:58.355771 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:58.356100 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:58.356105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:58.356269 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:58.356368 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.356374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.357226 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.357230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.357814 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.357823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.358251 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.358255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.358331 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.358774 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:58.358785 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:58.360057 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:58.362025 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.362027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.362028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.362201 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.362550 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.362555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.362560 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.362638 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.363024 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:58.363027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:58.363115 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 20:17:58.363620 (+ 505us) client_negotiation.cc:240] Negotiation successful
0620 20:17:58.363730 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 20:17:58.366134 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.356089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.24.20.189:44769)
0620 20:17:58.356189 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.356192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.356232 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:17:58.356346 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.356348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.356376 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.356475 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.357108 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.357591 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.358546 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.358854 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.358929 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.361994 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.361999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.362005 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.362030 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.364093 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.364547 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.364550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.364554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.364716 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.365021 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.365024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.365026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.365241 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.365457 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.365610 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.365978 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 20:17:58 dist-test-slave-pflr krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986678, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.189@KRBTEST.COM for kudu/127.24.20.188@KRBTEST.COM
I20260620 20:17:58.367383 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b: Bootstrap starting.
I20260620 20:17:58.368288 14551 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } attempt: 3
I20260620 20:17:58.370263 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:17:58.370705 14702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:58.371037 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.361432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53706)
0620 20:17:58.361518 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.361533 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:58.361604 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:58.362392 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:58.362395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:58.362540 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:58.362623 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.362628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.364178 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.364184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.364936 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.364947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.365477 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.365482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.365575 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.366012 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:58.366023 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:58.367383 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:58.369544 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.369547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.369549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.369782 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.370097 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.370100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.370101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.370183 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.370629 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:58.370632 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:58.370714 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 20:17:58.370784 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:17:58.370894 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 20:17:58.371603 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.361502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.24.20.188:39519)
0620 20:17:58.362151 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.362166 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.362180 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:17:58.362214 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.362218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.362260 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.362361 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.362734 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.364038 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.365079 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.365355 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.365769 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.367643 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.367647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.367650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.367672 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.369426 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.369890 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.369893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.369895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.369975 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.370300 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.370306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.370310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.370520 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.371280 (+ 760us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.371347 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.371445 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":116,"threads_started":1}
I20260620 20:17:58.371606 14702 log.cc:824] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:58.371627 14525 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8c2047ee3d464c8dbec38472352d0e62"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37942
I20260620 20:17:58.373808 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b: No bootstrap required, opened a new log
Jun 20 20:17:58 dist-test-slave-pflr krb5kdc[14470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986677, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.188@KRBTEST.COM
I20260620 20:17:58.374538 14626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a: Bootstrap starting.
I20260620 20:17:58.377207 14626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a: Neither blocks nor log segments found. Creating new log.
I20260620 20:17:58.377275 14702 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } }
I20260620 20:17:58.377452 14702 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0072d1b8f9140fea698673ec677788b, State: Initialized, Role: FOLLOWER
I20260620 20:17:58.377694 14702 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } }
I20260620 20:17:58.378041 14626 log.cc:824] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:58.378082 14709 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } } }
I20260620 20:17:58.378192 14709 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:17:58.378496 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.368676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:39519 (local address 127.0.0.1:53720)
0620 20:17:58.369086 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 20:17:58.369100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:17:58.369197 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:17:58.369864 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:17:58.369868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:17:58.370097 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:17:58.370239 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.370249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.371160 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.371164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.372018 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:17:58.372028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.372728 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.372731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:17:58.372818 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.373278 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:17:58.373294 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:17:58.374773 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:17:58.376746 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.376749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.376751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.376998 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.377362 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:17:58.377365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:17:58.377367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:17:58.377479 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:17:58.377963 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:17:58.377966 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:17:58.378073 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 20:17:58.378175 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 20:17:58.378309 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341}
I20260620 20:17:58.378961 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.369479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53720 (local address 127.24.20.188:39519)
0620 20:17:58.369590 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.369596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.369608 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:17:58.369642 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.369644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.369683 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.369755 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.370357 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.371036 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.372139 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.372434 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.372564 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.374943 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.374947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.374949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.374971 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.376586 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.377130 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.377136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.377140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.377238 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.377602 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.377605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.377606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.377807 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.377900 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.378719 (+ 819us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.378817 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:17:58.378995 14702 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:58.380153 14626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a: No bootstrap required, opened a new log
I20260620 20:17:58.381592 14551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62: Bootstrap starting.
I20260620 20:17:58.383967 14551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62: Neither blocks nor log segments found. Creating new log.
W20260620 20:17:58.384310 14721 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:17:58.384387 14721 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:17:58.384976 14551 log.cc:824] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62: Log is configured to *not* fsync() on all Append() calls
I20260620 20:17:58.383029 14633 master_runner.cc:186] Error getting master registration for 127.24.20.190:34681: OK, instance_id { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" instance_seqno: 1781986677905987 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:17:58.386543 14633 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:17:58.386677 14626 raft_consensus.cc:348] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } }
I20260620 20:17:58.386899 14626 raft_consensus.cc:729] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca400f9df3544f3ab1e4173e951c43a, State: Initialized, Role: FOLLOWER
I20260620 20:17:58.387229 14626 consensus_queue.cc:260] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } }
I20260620 20:17:58.387382 14551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62: No bootstrap required, opened a new log
I20260620 20:17:58.387679 14724 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } } }
I20260620 20:17:58.387856 14724 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:17:58.388496 14626 sys_catalog.cc:564] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:58.391073 14551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } }
I20260620 20:17:58.391273 14551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2047ee3d464c8dbec38472352d0e62, State: Initialized, Role: FOLLOWER
I20260620 20:17:58.391592 14551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } }
I20260620 20:17:58.392095 14736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } } }
I20260620 20:17:58.392333 14736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:17:58.392760 14551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:17:58.392800 14525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 20:17:58.392904 14525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:58.394093 14525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } }
W20260620 20:17:58.394189 14735 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:17:58.394255 14735 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:17:58.391746 14558 master_runner.cc:186] Error getting master registration for 127.24.20.190:34681: OK, instance_id { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" instance_seqno: 1781986677905987 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:17:58.394979 14525 leader_election.cc:288] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [CANDIDATE]: Term 1 election: Requested vote from peers aca400f9df3544f3ab1e4173e951c43a (127.24.20.189:44769), b0072d1b8f9140fea698673ec677788b (127.24.20.188:39519)
I20260620 20:17:58.395040 14558 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:17:58.395539 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c2047ee3d464c8dbec38472352d0e62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aca400f9df3544f3ab1e4173e951c43a"
I20260620 20:17:58.395780 14600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:58.395884 14676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c2047ee3d464c8dbec38472352d0e62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b0072d1b8f9140fea698673ec677788b"
I20260620 20:17:58.396104 14676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:17:58.397480 14676 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c2047ee3d464c8dbec38472352d0e62 in term 1.
I20260620 20:17:58.397866 14501 leader_election.cc:302] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c2047ee3d464c8dbec38472352d0e62, b0072d1b8f9140fea698673ec677788b; no voters:
I20260620 20:17:58.398324 14736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:17:58.398521 14736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [term 1 LEADER]: Becoming Leader. State: Replica: 8c2047ee3d464c8dbec38472352d0e62, State: Running, Role: LEADER
I20260620 20:17:58.398729 14600 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c2047ee3d464c8dbec38472352d0e62 in term 1.
I20260620 20:17:58.398844 14736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } }
I20260620 20:17:58.401005 14753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c2047ee3d464c8dbec38472352d0e62. Latest consensus state: current_term: 1 leader_uuid: "8c2047ee3d464c8dbec38472352d0e62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } } }
I20260620 20:17:58.401124 14753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [sys.catalog]: This master's current role is: LEADER
W20260620 20:17:58.402508 14504 tablet.cc:2369] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:17:58.403201 14483 master_runner.cc:263] No leader master found from master 8c2047ee3d464c8dbec38472352d0e62
I20260620 20:17:58.403257 14483 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 20:17:58.404640 14748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:17:58.404716 14748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:17:58.405264 14755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:17:58.406167 14755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:17:58.411672 14676 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b [term 1 FOLLOWER]: Refusing update from remote peer 8c2047ee3d464c8dbec38472352d0e62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:58.412451 14600 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a [term 1 FOLLOWER]: Refusing update from remote peer 8c2047ee3d464c8dbec38472352d0e62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:17:58.413100 14753 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:58.417850 14709 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c2047ee3d464c8dbec38472352d0e62. Latest consensus state: current_term: 1 leader_uuid: "8c2047ee3d464c8dbec38472352d0e62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } } }
I20260620 20:17:58.417977 14709 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:17:58.418115 14753 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:17:58.421923 14724 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c2047ee3d464c8dbec38472352d0e62. Latest consensus state: current_term: 1 leader_uuid: "8c2047ee3d464c8dbec38472352d0e62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } } }
I20260620 20:17:58.422063 14724 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:17:58.423844 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.405103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.24.20.188:39519)
0620 20:17:58.405236 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.405241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.412877 (+ 7636us) server_negotiation.cc:407] Connection header received
0620 20:17:58.413061 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.413065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.413118 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.413189 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.414385 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.415153 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.417456 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.417811 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.417881 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.419920 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.419924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.419927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.419953 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.421733 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.422377 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.422381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.422383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.422476 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.422797 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.422801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.422803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.423036 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.423143 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.423556 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.423676 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:17:58.432139 14760 mvcc.cc:205] Tried to move back new op lower bound from 7299017434766004224 to 7299017434725208064. Current Snapshot: MvccSnapshot[applied={T|T < 7299017434766004224}]
I20260620 20:17:58.432518 14736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c2047ee3d464c8dbec38472352d0e62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } } }
I20260620 20:17:58.432675 14736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:58.433450 14709 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c2047ee3d464c8dbec38472352d0e62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } } }
I20260620 20:17:58.433584 14709 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:17:58.433708 14762 mvcc.cc:205] Tried to move back new op lower bound from 7299017434766004224 to 7299017434725208064. Current Snapshot: MvccSnapshot[applied={T|T < 7299017434766004224}]
I20260620 20:17:58.434839 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.405892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.24.20.189:44769)
0620 20:17:58.405987 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.405992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.413249 (+ 7257us) server_negotiation.cc:407] Connection header received
0620 20:17:58.413653 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.413657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.413713 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.413770 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.415927 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.416721 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.420688 (+ 3967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.421094 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.421182 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.429376 (+ 8194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.429382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.429387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.429417 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.431455 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.432762 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.432768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.432770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.432856 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.433464 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.433467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.433469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.433692 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.433803 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.434568 (+ 765us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.434688 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 20:17:58.435002 14753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c2047ee3d464c8dbec38472352d0e62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } } }
I20260620 20:17:58.435110 14753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62 [sys.catalog]: This master's current role is: LEADER
I20260620 20:17:58.436761 14724 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c2047ee3d464c8dbec38472352d0e62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c2047ee3d464c8dbec38472352d0e62" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 34681 } } peers { permanent_uuid: "aca400f9df3544f3ab1e4173e951c43a" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 44769 } } peers { permanent_uuid: "b0072d1b8f9140fea698673ec677788b" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 39519 } } }
I20260620 20:17:58.436894 14724 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:17:58.438737 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:58.405611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.24.20.190:34681)
0620 20:17:58.405713 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.405719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.413571 (+ 7852us) server_negotiation.cc:407] Connection header received
0620 20:17:58.414813 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.414816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.414859 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.414904 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.417767 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.418474 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.422105 (+ 3631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:58.422495 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:58.422569 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:58.429776 (+ 7207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:17:58.429782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:17:58.429788 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:17:58.429819 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:17:58.432261 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.434119 (+ 1858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.434123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.434124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.434209 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:17:58.437571 (+ 3362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:17:58.437574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:17:58.437576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:17:58.437788 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:17:58.437899 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:58.438469 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 20:17:58.438592 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 20:17:58.440032 14764 mvcc.cc:205] Tried to move back new op lower bound from 7299017434766004224 to 7299017434725208064. Current Snapshot: MvccSnapshot[applied={T|T < 7299017434766004224}]
I20260620 20:17:58.442143 14755 catalog_manager.cc:1347] Generated new cluster ID: 3556ad564eb2439eb1b377fa29517102
I20260620 20:17:58.442211 14755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:17:58.494073 14755 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:17:58.495450 14755 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:17:58.514791 14755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c2047ee3d464c8dbec38472352d0e62: Generated new TSK 0
I20260620 20:17:58.515712 14755 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:17:58.557161 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:58.554233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.24.20.189:44769)
0620 20:17:58.554359 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.554365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.555622 (+ 1257us) server_negotiation.cc:407] Connection header received
0620 20:17:58.556101 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.556105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.556166 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.556225 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.557023 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60560: BlockingRecv error: recv got EOF from 127.0.0.1:60560 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 20:17:58.558315 14554 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:58.553394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37968 (local address 127.24.20.190:34681)
0620 20:17:58.553525 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.553529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.556005 (+ 2476us) server_negotiation.cc:407] Connection header received
0620 20:17:58.557524 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.557527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.557578 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.557636 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.558194 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37968: BlockingRecv error: recv got EOF from 127.0.0.1:37968 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 20:17:58.559962 14705 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:58.553855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.24.20.188:39519)
0620 20:17:58.553968 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.553972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.558802 (+ 4830us) server_negotiation.cc:407] Connection header received
0620 20:17:58.559215 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.559218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.559272 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.559328 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.559848 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53734: BlockingRecv error: recv got EOF from 127.0.0.1:53734 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 20:17:58.564556 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:58.563019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.24.20.189:44769)
0620 20:17:58.563132 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.563137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.563583 (+ 446us) server_negotiation.cc:407] Connection header received
0620 20:17:58.563790 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.563793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.563868 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.563927 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.564459 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60574: BlockingRecv error: recv got EOF from 127.0.0.1:60574 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 20:17:58.569612 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:58.568041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.24.20.189:44769)
0620 20:17:58.568138 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.568143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.568712 (+ 569us) server_negotiation.cc:407] Connection header received
0620 20:17:58.568914 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.568917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.568998 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.569049 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.569524 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60584: BlockingRecv error: recv got EOF from 127.0.0.1:60584 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 20:17:58.575069 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:58.573501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.24.20.189:44769)
0620 20:17:58.573599 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.573604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.574143 (+ 539us) server_negotiation.cc:407] Connection header received
0620 20:17:58.574328 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.574331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.574378 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.574429 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.574983 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60594: BlockingRecv error: recv got EOF from 127.0.0.1:60594 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 20:17:58.582597 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:58.580928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.24.20.189:44769)
0620 20:17:58.581053 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.581057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.581497 (+ 440us) server_negotiation.cc:407] Connection header received
0620 20:17:58.581674 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.581677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.581722 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.581771 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.582480 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60604: BlockingRecv error: recv got EOF from 127.0.0.1:60604 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 20:17:58.596583 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:58.594854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.24.20.189:44769)
0620 20:17:58.595009 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.595015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.595629 (+ 614us) server_negotiation.cc:407] Connection header received
0620 20:17:58.595834 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.595840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.595899 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.595954 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.596474 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60620: BlockingRecv error: recv got EOF from 127.0.0.1:60620 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 20:17:58.616022 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:58.614747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.24.20.189:44769)
0620 20:17:58.614867 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.614871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.615125 (+ 254us) server_negotiation.cc:407] Connection header received
0620 20:17:58.615291 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.615294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.615350 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.615444 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.615895 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60628: BlockingRecv error: recv got EOF from 127.0.0.1:60628 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 20:17:58.651193 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:58.650143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.24.20.189:44769)
0620 20:17:58.650314 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.650320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.650390 (+ 70us) server_negotiation.cc:407] Connection header received
0620 20:17:58.650545 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.650548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.650596 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.650680 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.651096 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60630: BlockingRecv error: recv got EOF from 127.0.0.1:60630 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 20:17:58.718653 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:58.717546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.24.20.189:44769)
0620 20:17:58.717711 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.717715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.717732 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:17:58.717870 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.717873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.717918 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.717991 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.718459 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60644: BlockingRecv error: recv got EOF from 127.0.0.1:60644 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 20:17:58.850310 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:58.849010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.24.20.189:44769)
0620 20:17:58.849184 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:58.849190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:58.849307 (+ 117us) server_negotiation.cc:407] Connection header received
0620 20:17:58.849498 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:58.849501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:58.849556 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:58.849660 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:58.850156 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60646: BlockingRecv error: recv got EOF from 127.0.0.1:60646 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260620 20:17:59.109545 14631 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:17:59.108545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.24.20.189:44769)
0620 20:17:59.108708 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:59.108712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:59.108790 (+ 78us) server_negotiation.cc:407] Connection header received
0620 20:17:59.108906 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:59.108909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:59.108958 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:59.109051 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:17:59.109447 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60662: BlockingRecv error: recv got EOF from 127.0.0.1:60662 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:17:59.385828 14721 catalog_manager.cc:1259] Loaded cluster ID: 3556ad564eb2439eb1b377fa29517102
I20260620 20:17:59.385922 14721 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b: loading cluster ID for follower catalog manager: success
I20260620 20:17:59.388091 14721 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b: acquiring CA information for follower catalog manager: success
I20260620 20:17:59.389420 14721 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b0072d1b8f9140fea698673ec677788b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:17:59.395365 14735 catalog_manager.cc:1259] Loaded cluster ID: 3556ad564eb2439eb1b377fa29517102
I20260620 20:17:59.395454 14735 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a: loading cluster ID for follower catalog manager: success
I20260620 20:17:59.397001 14735 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a: acquiring CA information for follower catalog manager: success
I20260620 20:17:59.398034 14735 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aca400f9df3544f3ab1e4173e951c43a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:17:59.627892 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:59.623928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.24.20.189:44769)
0620 20:17:59.624212 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:59.624216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:59.624233 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:17:59.624294 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:59.624296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:59.624421 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:59.624528 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:17:59.624845 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:59.625374 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:59.626295 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:59.626520 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:59.626602 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:59.626928 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:17:59.627366 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:17:59.627461 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:59.627617 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 20:17:59.627669 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260620 20:17:59.635162 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:59.631382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.24.20.188:39519)
0620 20:17:59.631601 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:59.631604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:59.631617 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:17:59.631774 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:59.631777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:59.631887 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:59.631984 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:17:59.632268 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:59.632902 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:59.633770 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:59.633994 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:59.634076 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:59.634341 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:17:59.634778 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:17:59.634882 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:59.634997 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 20:17:59.635048 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260620 20:17:59.642931 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:59.638145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.24.20.189:44769)
0620 20:17:59.638262 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:59.638268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:59.638759 (+ 491us) server_negotiation.cc:407] Connection header received
0620 20:17:59.638948 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:59.638951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:59.639008 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:59.639090 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:17:59.639342 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:59.640108 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:59.641903 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:59.642117 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:59.642187 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:59.642456 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:17:59.642536 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:17:59.642608 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:59.642723 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 20:17:59.642786 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:17:59.643128 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:59.638148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.24.20.190:34681)
0620 20:17:59.638380 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:59.638385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:59.638631 (+ 246us) server_negotiation.cc:407] Connection header received
0620 20:17:59.638827 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:59.638830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:59.639004 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:59.639114 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:17:59.639568 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:59.640343 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:59.641380 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:59.641739 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:59.641840 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:59.642369 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:17:59.642725 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:17:59.642809 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:59.642916 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 20:17:59.642978 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260620 20:17:59.644040 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:17:59.638455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.24.20.188:39519)
0620 20:17:59.638573 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 20:17:59.638579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:17:59.638744 (+ 165us) server_negotiation.cc:407] Connection header received
0620 20:17:59.638974 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:17:59.638977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:17:59.639030 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:17:59.639119 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:17:59.639532 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:59.639994 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:59.641410 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:17:59.641739 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:17:59.641841 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:17:59.642551 (+ 710us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:17:59.642646 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:17:59.642715 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:17:59.643869 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0620 20:17:59.643922 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:17:59.649833 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 14483
I20260620 20:17:59.656169 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 14558
I20260620 20:17:59.661898 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 14633
2026-06-20T20:17:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3088 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:17:59 dist-test-slave-pflr krb5kdc[14850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:17:59 dist-test-slave-pflr krb5kdc[14850](info): set up 2 sockets
Jun 20 20:17:59 dist-test-slave-pflr krb5kdc[14850](info): commencing operation
krb5kdc: starting...
W20260620 20:18:00.693276 24658 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/test-user.keytab.
Jun 20 20:18:00 dist-test-slave-pflr krb5kdc[14850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:18:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:18:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.20.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
Entry for principal kudu/127.24.20.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab.
I20260620 20:18:00.816331 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:36127
--webserver_interface=127.24.20.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:36127,127.24.20.189:39433,127.24.20.188:42239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:18:00.933560 14863 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:18:00.933846 14863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:00.933897 14863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:00.933969 14863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:00.937916 14863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:00.937995 14863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:00.938066 14863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:00.938098 14863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:00.938133 14863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:00.943252 14863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:36127,127.24.20.189:39433,127.24.20.188:42239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.190.keytab
--principal=kudu/127.24.20.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.190:36127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.20.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:00.944247 14863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:00.945281 14863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:00.953456 14868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:00.953462 14871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:00.953464 14869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:00.953801 14863 server_base.cc:1034] running on GCE node
I20260620 20:18:00.954396 14863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:00.955437 14863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:00.956666 14863 hybrid_clock.cc:648] HybridClock initialized: now 1781986680956649 us; error 29 us; skew 500 ppm
Jun 20 20:18:00 dist-test-slave-pflr krb5kdc[14850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986680, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:00.959429 14863 init.cc:377] Logged in from keytab as kudu/127.24.20.190@KRBTEST.COM (short username kudu)
I20260620 20:18:00.960290 14863 webserver.cc:460] Webserver started at http://127.24.20.190:41545/ using document root <none> and password file <none>
I20260620 20:18:00.960893 14863 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:00.960994 14863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:00.961215 14863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:00.963202 14863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/data/instance:
uuid: "fcb9cd11b8f14c779e8898ae366b225b"
format_stamp: "Formatted at 2026-06-20 20:18:00 on dist-test-slave-pflr"
server_key: "742363b510328c2e337b67d91e81373c"
server_key_iv: "082f280599bfcfabb053bdbab471b5d7"
server_key_version: "encryptionkey@0"
I20260620 20:18:00.963768 14863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/wal/instance:
uuid: "fcb9cd11b8f14c779e8898ae366b225b"
format_stamp: "Formatted at 2026-06-20 20:18:00 on dist-test-slave-pflr"
server_key: "742363b510328c2e337b67d91e81373c"
server_key_iv: "082f280599bfcfabb053bdbab471b5d7"
server_key_version: "encryptionkey@0"
I20260620 20:18:00.967469 14863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 20:18:00.969877 14878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:00.971078 14863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 20:18:00.971216 14863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/wal
uuid: "fcb9cd11b8f14c779e8898ae366b225b"
format_stamp: "Formatted at 2026-06-20 20:18:00 on dist-test-slave-pflr"
server_key: "742363b510328c2e337b67d91e81373c"
server_key_iv: "082f280599bfcfabb053bdbab471b5d7"
server_key_version: "encryptionkey@0"
I20260620 20:18:00.971326 14863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:00.983999 14863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:00.986774 14863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:00.986980 14863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:00.993695 14863 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.190:36127
I20260620 20:18:00.993995 14930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.190:36127 every 8 connection(s)
I20260620 20:18:00.994630 14863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/data/info.pb
I20260620 20:18:00.997340 14931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:01.001060 14931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:18:01.002264 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 14863
I20260620 20:18:01.002426 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-0/wal/instance
I20260620 20:18:01.002717 24658 external_mini_cluster.cc:1428] Setting key 5e09499f3a18a60419514df334ab1d16
WARNING: no policy specified for kudu/127.24.20.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.189@KRBTEST.COM" created.
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986680, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:18:01.020289 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.003446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.24.20.190:36127)
0620 20:18:01.004462 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.004474 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.004513 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:18:01.005405 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.005414 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.005663 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.005852 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.006777 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.008313 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.009395 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.010022 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.010202 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.015218 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.015224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.015233 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.015254 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.017026 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.017654 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.017659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.017660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.017829 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.018253 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.018255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.018256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.018624 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.018728 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.019271 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.019501 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":905,"thread_start_us":151,"threads_started":1}
I20260620 20:18:01.020291 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.003446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:36127 (local address 127.0.0.1:38620)
0620 20:18:01.004046 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.004111 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:01.005089 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:01.005885 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:01.005890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:01.006230 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:01.006603 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.006618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.008454 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.008457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.009261 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.009270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.010182 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.010184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.010276 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.010871 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:01.010886 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:01.015060 (+ 4174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:01.017138 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.017151 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.017164 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.017497 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.017981 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.017987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.017990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.018145 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.018759 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:01.018768 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:01.018956 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 20:18:01.019171 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 20:18:01.019610 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":125,"spinlock_wait_cycles":56704,"thread_start_us":151,"threads_started":1}
I20260620 20:18:01.022746 14931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:18:01.023504 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.023244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39433 (local address 127.0.0.1:36454)
0620 20:18:01.023369 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.023443 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 20:18:01.023684 14880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.20.189:39433: connect: Connection refused (error 111)
I20260620 20:18:01.024915 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.024435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39433 (local address 127.0.0.1:36460)
0620 20:18:01.024557 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.024862 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 20:18:01.025085 14931 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.189:39433: Network error: Client connection negotiation failed: client connection to 127.24.20.189:39433: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.189.keytab.
Entry for principal kudu/127.24.20.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.189.keytab.
I20260620 20:18:01.031342 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.189:39433
--webserver_interface=127.24.20.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.189.keytab
--principal=kudu/127.24.20.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:36127,127.24.20.189:39433,127.24.20.188:42239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:18:01.074321 14931 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } attempt: 1
I20260620 20:18:01.075273 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.074921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39433 (local address 127.0.0.1:36476)
0620 20:18:01.075106 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.075183 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 20:18:01.075979 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.075743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39433 (local address 127.0.0.1:36482)
0620 20:18:01.075853 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.075906 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 20:18:01.076114 14931 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.189:39433: Network error: Client connection negotiation failed: client connection to 127.24.20.189:39433: connect: Connection refused (error 111)
W20260620 20:18:01.143146 14938 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:18:01.143491 14938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:01.143572 14938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:01.143656 14938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 20:18:01.144301 14931 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } attempt: 2
I20260620 20:18:01.145220 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.144894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39433 (local address 127.0.0.1:36486)
0620 20:18:01.145074 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.145143 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 20:18:01.146018 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.145777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39433 (local address 127.0.0.1:36500)
0620 20:18:01.145907 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.145959 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.189:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 20:18:01.146162 14931 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.189:39433: Network error: Client connection negotiation failed: client connection to 127.24.20.189:39433: connect: Connection refused (error 111)
W20260620 20:18:01.148032 14938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:01.148084 14938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:01.148103 14938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:01.148121 14938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:01.148161 14938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:01.153524 14938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:36127,127.24.20.189:39433,127.24.20.188:42239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.189.keytab
--principal=kudu/127.24.20.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.189:39433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.20.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:01.154621 14938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:01.155647 14938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:18:01.164723 14944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:01.164723 14946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:01.164769 14943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:01.164906 14938 server_base.cc:1034] running on GCE node
I20260620 20:18:01.165620 14938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:01.166778 14938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:01.167948 14938 hybrid_clock.cc:648] HybridClock initialized: now 1781986681167916 us; error 41 us; skew 500 ppm
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:01.171052 14938 init.cc:377] Logged in from keytab as kudu/127.24.20.189@KRBTEST.COM (short username kudu)
I20260620 20:18:01.171983 14938 webserver.cc:460] Webserver started at http://127.24.20.189:42987/ using document root <none> and password file <none>
I20260620 20:18:01.172573 14938 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:01.172735 14938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:01.172981 14938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:01.175189 14938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/data/instance:
uuid: "ed0ad437c2f24085b8a7b25815786480"
format_stamp: "Formatted at 2026-06-20 20:18:01 on dist-test-slave-pflr"
server_key: "160bc29124a55d87cde059a2d5208ca9"
server_key_iv: "4dcab1d3c60e259f5033718f21318f3c"
server_key_version: "encryptionkey@0"
I20260620 20:18:01.175800 14938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/wal/instance:
uuid: "ed0ad437c2f24085b8a7b25815786480"
format_stamp: "Formatted at 2026-06-20 20:18:01 on dist-test-slave-pflr"
server_key: "160bc29124a55d87cde059a2d5208ca9"
server_key_iv: "4dcab1d3c60e259f5033718f21318f3c"
server_key_version: "encryptionkey@0"
I20260620 20:18:01.179667 14938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 20:18:01.182255 14953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:01.183688 14938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:18:01.183791 14938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/wal
uuid: "ed0ad437c2f24085b8a7b25815786480"
format_stamp: "Formatted at 2026-06-20 20:18:01 on dist-test-slave-pflr"
server_key: "160bc29124a55d87cde059a2d5208ca9"
server_key_iv: "4dcab1d3c60e259f5033718f21318f3c"
server_key_version: "encryptionkey@0"
I20260620 20:18:01.183877 14938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:01.195307 14938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:01.198686 14938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:01.198835 14938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:01.205488 14938 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.189:39433
I20260620 20:18:01.205771 15005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.189:39433 every 8 connection(s)
I20260620 20:18:01.206453 14938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/data/info.pb
I20260620 20:18:01.207370 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 14938
I20260620 20:18:01.207505 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-1/wal/instance
I20260620 20:18:01.207819 24658 external_mini_cluster.cc:1428] Setting key 3c21e8bb0e8f77ade7ca7388ff0aa683
I20260620 20:18:01.209856 15006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:01.214335 15006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.20.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.20.188@KRBTEST.COM" created.
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.189@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:18:01.228657 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.216687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.24.20.190:36127)
0620 20:18:01.216841 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.216844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.217760 (+ 916us) server_negotiation.cc:407] Connection header received
0620 20:18:01.218735 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.218738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.218794 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.218906 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.220205 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.220822 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.222090 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.222330 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.222470 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.224962 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.224966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.224973 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.225005 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.226499 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.227015 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.227019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.227021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.227096 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.227440 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.227445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.227447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.227626 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.227738 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.228316 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.228445 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:18:01.229511 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.217060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:36127 (local address 127.0.0.1:38626)
0620 20:18:01.217598 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.217640 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:01.218520 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:01.219062 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:01.219069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:01.219410 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:01.220053 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.220068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.220999 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.221005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.221952 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.221962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.222521 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.222526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.222703 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.223316 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:01.223331 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:01.224793 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:01.226622 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.226628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.226643 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.226898 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.227201 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.227204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.227206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.227340 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.227767 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:01.227774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:01.228062 (+ 288us) client_negotiation.cc:769] Sending connection context
0620 20:18:01.228253 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 20:18:01.228673 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":119,"threads_started":1}
I20260620 20:18:01.230935 15006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.20.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.188.keytab.
Entry for principal kudu/127.24.20.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.188.keytab.
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.189@KRBTEST.COM for kudu/127.24.20.189@KRBTEST.COM
I20260620 20:18:01.236236 24658 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
/tmp/dist-test-taskVWkq7u/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.188:42239
--webserver_interface=127.24.20.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.188.keytab
--principal=kudu/127.24.20.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.20.148:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.20.190:36127,127.24.20.189:39433,127.24.20.188:42239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:18:01.237327 14931 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } attempt: 3
I20260620 20:18:01.240562 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.231426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39433 (local address 127.0.0.1:36502)
0620 20:18:01.231548 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.231560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:01.231626 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:01.232312 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:01.232315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:01.232477 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:01.232569 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.232574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.233552 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.233555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.234068 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.234074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.234669 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.234672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.234763 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.235258 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:01.235270 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:01.236584 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:01.238833 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.238838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.238840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.239095 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.239418 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.239424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.239436 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.239535 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.240064 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:01.240067 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:01.240157 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 20:18:01.240266 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 20:18:01.240394 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:18:01.241149 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.231583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.24.20.189:39433)
0620 20:18:01.231910 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.231921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.231944 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:18:01.231987 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.231995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.232186 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.232336 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.232700 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.233418 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.234184 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.234556 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.234674 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.236755 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.236763 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.236774 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.236793 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.238716 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.239220 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.239226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.239228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.239312 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.239640 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.239646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.239650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.239943 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.240078 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.240381 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.240569 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":117,"threads_started":1}
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986680, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.189@KRBTEST.COM
I20260620 20:18:01.243263 15006 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:18:01.243975 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.243718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34236)
0620 20:18:01.243833 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.243914 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 20:18:01.244249 14955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
I20260620 20:18:01.245270 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.244983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34248)
0620 20:18:01.245138 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.245192 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 20:18:01.245386 15006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:42239: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
I20260620 20:18:01.246256 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.237718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39433 (local address 127.0.0.1:36516)
0620 20:18:01.237885 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.237904 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:01.237991 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:01.238494 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:01.238497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:01.238676 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:01.238789 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.238799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.239628 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.239633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.240287 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.240296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.241113 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.241117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.241228 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.241648 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:01.241658 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:01.242822 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:01.244883 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.244886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.244888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.245076 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.245335 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.245339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.245341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.245431 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.245886 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:01.245888 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:01.245966 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 20:18:01.246055 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 20:18:01.246149 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 20:18:01.246528 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.237928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.24.20.189:39433)
0620 20:18:01.238243 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.238254 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.238268 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:18:01.238303 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.238307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.238352 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.238453 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.238879 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.239529 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.240405 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.240986 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.241113 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.242983 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.242986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.242988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.243008 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.244757 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.245173 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.245175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.245176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.245238 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.245534 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.245540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.245545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.245776 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.245893 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.246148 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.246279 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
I20260620 20:18:01.247175 14931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:18:01.247828 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.247623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34252)
0620 20:18:01.247729 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.247774 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 20:18:01.248544 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.248326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34254)
0620 20:18:01.248457 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.248495 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 20:18:01.248709 14931 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:42239: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
I20260620 20:18:01.279884 14931 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } attempt: 1
I20260620 20:18:01.280717 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.280404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34260)
0620 20:18:01.280550 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.280639 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 20:18:01.281454 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.281246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34276)
0620 20:18:01.281353 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.281397 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 20:18:01.281610 14931 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:42239: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
I20260620 20:18:01.294629 15006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } attempt: 1
I20260620 20:18:01.295403 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.295121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34280)
0620 20:18:01.295283 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.295337 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 20:18:01.296118 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.295933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34292)
0620 20:18:01.296030 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.296072 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 20:18:01.296311 15006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:42239: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
W20260620 20:18:01.349462 15012 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:18:01.349726 15012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:18:01.349748 15012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:18:01.349836 15012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:18:01.353822 15012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:18:01.353869 15012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:18:01.353888 15012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:18:01.353906 15012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:18:01.353924 15012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:18:01.356763 14931 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } attempt: 2
I20260620 20:18:01.357587 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.357277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34302)
0620 20:18:01.357418 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.357485 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 20:18:01.358253 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.358053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34318)
0620 20:18:01.358158 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.358205 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 20:18:01.358368 14931 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:42239: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
I20260620 20:18:01.359267 15012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.20.148:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.20.190:36127,127.24.20.189:39433,127.24.20.188:42239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/krb5kdc/kudu_127.24.20.188.keytab
--principal=kudu/127.24.20.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.20.188:42239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.20.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 19:21:02 UTC on ccycloud.quasar-kkgvxp.root.comops.site
build id 4457675
I20260620 20:18:01.360244 15012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:18:01.361274 15012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:18:01.364478 15006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } attempt: 2
I20260620 20:18:01.365222 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.364921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34326)
0620 20:18:01.365092 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.365152 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 20:18:01.365938 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.365722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34336)
0620 20:18:01.365850 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.365888 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 20:18:01.366088 15006 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.20.188:42239: Network error: Client connection negotiation failed: client connection to 127.24.20.188:42239: connect: Connection refused (error 111)
W20260620 20:18:01.370070 15020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:01.370117 15012 server_base.cc:1034] running on GCE node
W20260620 20:18:01.370064 15019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:18:01.370064 15022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:18:01.371044 15012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:18:01.372196 15012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:18:01.373414 15012 hybrid_clock.cc:648] HybridClock initialized: now 1781986681373400 us; error 34 us; skew 500 ppm
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:18:01.376345 15012 init.cc:377] Logged in from keytab as kudu/127.24.20.188@KRBTEST.COM (short username kudu)
I20260620 20:18:01.377204 15012 webserver.cc:460] Webserver started at http://127.24.20.188:36335/ using document root <none> and password file <none>
I20260620 20:18:01.377705 15012 fs_manager.cc:359] Metadata directory not provided
I20260620 20:18:01.377764 15012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:18:01.377916 15012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:18:01.379829 15012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/data/instance:
uuid: "5ae4729ab354429d8def72f4c76eab0f"
format_stamp: "Formatted at 2026-06-20 20:18:01 on dist-test-slave-pflr"
server_key: "377b965549d9011a7d8421ec143f1c31"
server_key_iv: "f81dd806d04f12b774aff6312a3beb56"
server_key_version: "encryptionkey@0"
I20260620 20:18:01.380324 15012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/wal/instance:
uuid: "5ae4729ab354429d8def72f4c76eab0f"
format_stamp: "Formatted at 2026-06-20 20:18:01 on dist-test-slave-pflr"
server_key: "377b965549d9011a7d8421ec143f1c31"
server_key_iv: "f81dd806d04f12b774aff6312a3beb56"
server_key_version: "encryptionkey@0"
I20260620 20:18:01.383942 15012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 20:18:01.386397 15029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:18:01.387725 15012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 20:18:01.387856 15012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/data,/tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/wal
uuid: "5ae4729ab354429d8def72f4c76eab0f"
format_stamp: "Formatted at 2026-06-20 20:18:01 on dist-test-slave-pflr"
server_key: "377b965549d9011a7d8421ec143f1c31"
server_key_iv: "f81dd806d04f12b774aff6312a3beb56"
server_key_version: "encryptionkey@0"
I20260620 20:18:01.387960 15012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:18:01.400439 15012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:18:01.403929 15012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:18:01.404145 15012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:18:01.411355 15012 rpc_server.cc:307] RPC server started. Bound to: 127.24.20.188:42239
I20260620 20:18:01.411695 15081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.20.188:42239 every 8 connection(s)
I20260620 20:18:01.412304 15012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/data/info.pb
I20260620 20:18:01.413492 24658 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu as pid 15012
I20260620 20:18:01.413622 24658 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVWkq7u/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986405628821-24658-0/minicluster-data/master-2/wal/instance
I20260620 20:18:01.413944 24658 external_mini_cluster.cc:1428] Setting key 1d51bc7f63f32b3057ae0bc63e15361b
I20260620 20:18:01.415657 15082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:18:01.420225 15082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.188@KRBTEST.COM
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.188@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
I20260620 20:18:01.429704 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.415345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.24.20.188:42239)
0620 20:18:01.415757 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.415786 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.415826 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:18:01.416688 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.416710 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.417122 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.417500 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.418168 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.419654 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.420497 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.421178 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.421330 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.424434 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.424445 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.424458 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.424492 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.426385 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.426868 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.426874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.426878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.426976 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.427288 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.427294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.427298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.427790 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.427927 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.428307 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.428586 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":101,"threads_started":1}
I20260620 20:18:01.432358 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.422217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38636 (local address 127.24.20.190:36127)
0620 20:18:01.422370 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.422375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.422774 (+ 399us) server_negotiation.cc:407] Connection header received
0620 20:18:01.422929 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.422931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.422974 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.423043 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.423773 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.424377 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.425315 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.425642 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.425760 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.428259 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.428263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.428266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.428289 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.430526 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.431011 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.431014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.431015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.431084 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.431404 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.431407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.431408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.431589 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.431692 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.432076 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.432186 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:18:01.432356 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.422127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.190:36127 (local address 127.0.0.1:38636)
0620 20:18:01.422618 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.422677 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:01.422815 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:01.423135 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:01.423142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:01.423516 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:01.423637 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.423643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.424513 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.424516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.425189 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.425198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.425763 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.425766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.425849 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.426347 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:01.426379 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:01.428126 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:01.430634 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.430642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.430646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.430906 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.431175 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.431178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.431179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.431279 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.431725 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:01.431730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:01.431841 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 20:18:01.431990 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 20:18:01.432162 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"spinlock_wait_cycles":3456,"thread_start_us":84,"threads_started":1}
I20260620 20:18:01.433627 15082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.190@KRBTEST.COM
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.188@KRBTEST.COM for kudu/127.24.20.189@KRBTEST.COM
I20260620 20:18:01.441550 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.434083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.189:39433 (local address 127.0.0.1:36526)
0620 20:18:01.434219 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.434232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:01.434329 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:01.434630 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:01.434633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:01.434795 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:01.434875 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.434880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.435624 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.435630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.436130 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.436138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.436633 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.436637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.436730 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.437168 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:01.437182 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:01.438509 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:01.440098 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.440104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.440107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.440296 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.440582 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.440587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.440589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.440695 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.441072 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:01.441075 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:01.441160 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 20:18:01.441271 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:18:01.441390 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 20:18:01.441582 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.434260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.24.20.189:39433)
0620 20:18:01.434409 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.434413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.434426 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:18:01.434473 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.434475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.434517 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.434618 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.435007 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.435516 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.436252 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.436496 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.436639 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.438649 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.438652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.438654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.438675 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.439964 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.440399 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.440404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.440406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.440485 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.440796 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.440799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.440801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.440967 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.441083 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.441332 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.441447 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:18:01.441620 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.432250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.24.20.190:36127)
0620 20:18:01.432587 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.432619 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.432634 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:18:01.432674 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.432678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.432724 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.432795 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.433484 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.434360 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.435145 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.435405 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.435520 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.438238 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.438242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.438245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.438273 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.439944 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.440376 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.440380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.440381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.440462 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.440773 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.440777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.440779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.440955 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.441160 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.441299 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.441419 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260620 20:18:01.442417 15082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.188@KRBTEST.COM for kudu/127.24.20.188@KRBTEST.COM
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.20.189@KRBTEST.COM
I20260620 20:18:01.451017 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.442899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34362)
0620 20:18:01.443077 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.443091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:01.443163 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:01.443352 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:01.443355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:01.443533 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:01.443590 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.443594 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.444291 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.444297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.444875 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.444883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.445334 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.445338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.445427 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.445899 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:01.445914 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:01.447318 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:01.449319 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.449322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.449324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.449579 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.449890 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.449895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.449898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.450039 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.450527 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:01.450533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:01.450625 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 20:18:01.450726 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:18:01.450841 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":4}
I20260620 20:18:01.451558 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.443020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34362 (local address 127.24.20.188:42239)
0620 20:18:01.443176 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.443182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.443197 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:18:01.443230 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.443233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.443265 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.443330 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.443699 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.444187 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.444985 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.445228 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.445338 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.447468 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.447471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.447474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.447498 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.449174 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.449701 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.449704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.449705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.449784 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.450178 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.450181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.450182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.450405 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.450510 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.451331 (+ 821us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.451436 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":103}
I20260620 20:18:01.452044 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.442968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.24.20.189:39433)
0620 20:18:01.443078 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.443083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.443103 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:18:01.443243 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.443247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.443285 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.443378 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.443941 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.444404 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.445226 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.445457 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.445568 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.448200 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.448204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.448206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.448230 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.450072 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.450559 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.450562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.450564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.450646 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.450957 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.450960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.450961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.451181 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.451364 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.451804 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.451897 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:18:01.455035 15082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f: Bootstrap starting.
I20260620 20:18:01.455313 24658 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:18:01.457261 15006 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } attempt: 3
I20260620 20:18:01.456404 14905 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fcb9cd11b8f14c779e8898ae366b225b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38644
I20260620 20:18:01.457579 15082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:01.457758 14931 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } attempt: 3
I20260620 20:18:01.458461 15082 log.cc:824] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:01.461050 15082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f: No bootstrap required, opened a new log
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986680, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.190@KRBTEST.COM for kudu/127.24.20.188@KRBTEST.COM
Jun 20 20:18:01 dist-test-slave-pflr krb5kdc[14850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986681, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.20.189@KRBTEST.COM for kudu/127.24.20.188@KRBTEST.COM
I20260620 20:18:01.465389 15082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } }
I20260620 20:18:01.465683 15082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae4729ab354429d8def72f4c76eab0f, State: Initialized, Role: FOLLOWER
I20260620 20:18:01.466056 15082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } }
I20260620 20:18:01.467530 15082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:01.468451 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.457654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34370)
0620 20:18:01.458018 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.458033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:01.458113 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:01.458650 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:01.458653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:01.458837 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:01.458940 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.458946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.459987 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.459993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.460852 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.460861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.461540 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.461544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.461635 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.462367 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:01.462386 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:01.464306 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:01.466744 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.466747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.466750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.466997 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.467280 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.467286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.467288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.467400 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.467903 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:01.467906 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:01.468022 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 20:18:01.468136 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 20:18:01.468276 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283}
I20260620 20:18:01.468685 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.458125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.20.188:42239 (local address 127.0.0.1:34378)
0620 20:18:01.458221 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 20:18:01.458236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:18:01.458336 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:18:01.458870 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:18:01.458874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:18:01.459082 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:18:01.459197 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.459208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.460055 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.460058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.460812 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:18:01.460823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.461526 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.461530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:18:01.461612 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.462273 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:18:01.462287 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:18:01.464726 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:18:01.466964 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.466968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.466969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.467224 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.467552 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:18:01.467555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:18:01.467557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:18:01.467642 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:18:01.468192 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:18:01.468195 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:18:01.468290 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 20:18:01.468368 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 20:18:01.468504 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 20:18:01.468884 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.457733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.24.20.188:42239)
0620 20:18:01.457913 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.457919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.458441 (+ 522us) server_negotiation.cc:407] Connection header received
0620 20:18:01.458480 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.458483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.458525 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.459056 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.459137 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.459866 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.460974 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.461419 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.461857 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.464447 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.464451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.464453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.464481 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.466635 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.467104 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.467107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.467109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.467188 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.467491 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.467494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.467496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.467762 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.467901 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.468202 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.468322 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":44}
I20260620 20:18:01.469365 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.458248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.24.20.188:42239)
0620 20:18:01.458623 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.458632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.458648 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:18:01.458681 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.458684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.458732 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.459319 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.459419 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.459905 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.460952 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.461413 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.461518 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.464877 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.464883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.464889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.464918 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.466849 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.467333 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.467338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.467343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.467425 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.467756 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.467761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.467762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.468080 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.468951 (+ 871us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.469022 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.469147 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":306,"thread_start_us":127,"threads_started":1}
I20260620 20:18:01.471977 15006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480: Bootstrap starting.
I20260620 20:18:01.472257 14931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b: Bootstrap starting.
I20260620 20:18:01.474578 15006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:01.474653 14931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b: Neither blocks nor log segments found. Creating new log.
I20260620 20:18:01.475337 15006 log.cc:824] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:01.475416 14931 log.cc:824] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:18:01.470691 15090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } } }
I20260620 20:18:01.476459 15090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f [sys.catalog]: This master's current role is: FOLLOWER
W20260620 20:18:01.473347 15101 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:18:01.476670 15101 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:18:01.477649 14931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b: No bootstrap required, opened a new log
I20260620 20:18:01.478159 15006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480: No bootstrap required, opened a new log
I20260620 20:18:01.477378 15012 master_runner.cc:186] Error getting master registration for 127.24.20.190:36127: OK, instance_id { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" instance_seqno: 1781986680991804 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:18:01.479857 15012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:18:01.480628 14931 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } }
I20260620 20:18:01.480839 14931 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb9cd11b8f14c779e8898ae366b225b, State: Initialized, Role: FOLLOWER
I20260620 20:18:01.481158 14931 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } }
I20260620 20:18:01.481674 15105 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } } }
I20260620 20:18:01.481808 15105 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:18:01.481873 15006 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } }
I20260620 20:18:01.482087 15006 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0ad437c2f24085b8a7b25815786480, State: Initialized, Role: FOLLOWER
I20260620 20:18:01.482501 14931 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:01.482429 15006 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } }
I20260620 20:18:01.482584 14905 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 20:18:01.482695 14905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:01.483695 15006 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:18:01.484398 14905 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } }
I20260620 20:18:01.485036 15106 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } } }
I20260620 20:18:01.485255 15106 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:18:01.487951 14980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcb9cd11b8f14c779e8898ae366b225b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ed0ad437c2f24085b8a7b25815786480"
W20260620 20:18:01.488091 15122 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:18:01.488157 15122 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:18:01.488149 14980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480 [term 0 FOLLOWER]: Advancing to term 1
W20260620 20:18:01.488382 14884 tablet.cc:2369] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:18:01.489490 14980 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcb9cd11b8f14c779e8898ae366b225b in term 1.
I20260620 20:18:01.489951 14880 leader_election.cc:302] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ed0ad437c2f24085b8a7b25815786480, fcb9cd11b8f14c779e8898ae366b225b; no voters:
I20260620 20:18:01.490175 14905 leader_election.cc:288] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [CANDIDATE]: Term 1 election: Requested vote from peers ed0ad437c2f24085b8a7b25815786480 (127.24.20.189:39433), 5ae4729ab354429d8def72f4c76eab0f (127.24.20.188:42239)
I20260620 20:18:01.490610 15056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcb9cd11b8f14c779e8898ae366b225b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5ae4729ab354429d8def72f4c76eab0f"
I20260620 20:18:01.490835 15056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:18:01.491932 15056 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcb9cd11b8f14c779e8898ae366b225b in term 1.
I20260620 20:18:01.494722 14863 master_runner.cc:263] No leader master found from master fcb9cd11b8f14c779e8898ae366b225b
I20260620 20:18:01.494793 14863 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:18:01.495005 14938 master_runner.cc:263] No leader master found from master ed0ad437c2f24085b8a7b25815786480
I20260620 20:18:01.495091 14938 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:18:01.495213 15105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:18:01.495420 15105 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [term 1 LEADER]: Becoming Leader. State: Replica: fcb9cd11b8f14c779e8898ae366b225b, State: Running, Role: LEADER
I20260620 20:18:01.495733 15105 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } }
I20260620 20:18:01.497798 15135 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcb9cd11b8f14c779e8898ae366b225b. Latest consensus state: current_term: 1 leader_uuid: "fcb9cd11b8f14c779e8898ae366b225b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } } }
I20260620 20:18:01.497931 15135 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:01.498298 15138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:18:01.499186 15138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 20:18:01.499554 15128 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:18:01.499634 15128 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:18:01.511055 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.494797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.24.20.190:36127)
0620 20:18:01.494921 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.494925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.501537 (+ 6612us) server_negotiation.cc:407] Connection header received
0620 20:18:01.501735 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.501738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.501795 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.501850 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.502910 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.503753 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.504986 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.505361 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.505439 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.507356 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.507360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.507363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.507389 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.509212 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.509703 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.509708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.509711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.509795 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.510115 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.510118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.510120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.510323 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.510421 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.510753 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.510880 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:18:01.518189 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.500954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.24.20.188:42239)
0620 20:18:01.501058 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.501062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.508864 (+ 7802us) server_negotiation.cc:407] Connection header received
0620 20:18:01.509055 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.509059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.509120 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.509183 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.510058 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.510790 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.512064 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.512401 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.512476 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.514345 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.514351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.514355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.514388 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.516189 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.516704 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.516708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.516710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.516794 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.517112 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.517116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.517118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.517330 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.517457 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.517816 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.517974 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:18:01.523134 15056 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f [term 1 FOLLOWER]: Refusing update from remote peer fcb9cd11b8f14c779e8898ae366b225b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:01.523984 15135 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:01.525048 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:01.500235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.24.20.189:39433)
0620 20:18:01.500349 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.500354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.504845 (+ 4491us) server_negotiation.cc:407] Connection header received
0620 20:18:01.505015 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.505019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.505065 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.505114 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.506035 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.506791 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.518807 (+ 12016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:01.519168 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:01.519255 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:01.521068 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:18:01.521072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:18:01.521075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:18:01.521103 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:18:01.522856 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.523378 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.523382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.523384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.523467 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:18:01.523779 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:18:01.523782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:18:01.523784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:18:01.524016 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:18:01.524152 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:01.524764 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 20:18:01.524885 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:18:01.527918 15090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcb9cd11b8f14c779e8898ae366b225b. Latest consensus state: current_term: 1 leader_uuid: "fcb9cd11b8f14c779e8898ae366b225b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } } }
I20260620 20:18:01.528025 15090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:18:01.529330 14979 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480 [term 1 FOLLOWER]: Refusing update from remote peer fcb9cd11b8f14c779e8898ae366b225b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:18:01.529870 15135 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:18:01.533746 15106 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcb9cd11b8f14c779e8898ae366b225b. Latest consensus state: current_term: 1 leader_uuid: "fcb9cd11b8f14c779e8898ae366b225b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } } }
I20260620 20:18:01.533874 15106 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:18:01.536342 15105 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcb9cd11b8f14c779e8898ae366b225b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } } }
I20260620 20:18:01.536479 15105 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:01.536985 15090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcb9cd11b8f14c779e8898ae366b225b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } } }
I20260620 20:18:01.537114 15090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:18:01.537477 15140 mvcc.cc:205] Tried to move back new op lower bound from 7299017447507623936 to 7299017447409618944. Current Snapshot: MvccSnapshot[applied={T|T < 7299017447507623936}]
I20260620 20:18:01.540958 15142 mvcc.cc:205] Tried to move back new op lower bound from 7299017447507623936 to 7299017447409618944. Current Snapshot: MvccSnapshot[applied={T|T < 7299017447507623936 or (T in {7299017447507623936})}]
I20260620 20:18:01.543038 15135 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcb9cd11b8f14c779e8898ae366b225b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } } }
I20260620 20:18:01.543162 15135 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b [sys.catalog]: This master's current role is: LEADER
I20260620 20:18:01.543931 15106 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcb9cd11b8f14c779e8898ae366b225b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcb9cd11b8f14c779e8898ae366b225b" member_type: VOTER last_known_addr { host: "127.24.20.190" port: 36127 } } peers { permanent_uuid: "ed0ad437c2f24085b8a7b25815786480" member_type: VOTER last_known_addr { host: "127.24.20.189" port: 39433 } } peers { permanent_uuid: "5ae4729ab354429d8def72f4c76eab0f" member_type: VOTER last_known_addr { host: "127.24.20.188" port: 42239 } } }
I20260620 20:18:01.544057 15106 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:18:01.546505 15144 mvcc.cc:205] Tried to move back new op lower bound from 7299017447507623936 to 7299017447409618944. Current Snapshot: MvccSnapshot[applied={T|T < 7299017447507623936}]
I20260620 20:18:01.550108 15138 catalog_manager.cc:1347] Generated new cluster ID: e5d41a4cf9c2452594d8a04a2bcf7cef
I20260620 20:18:01.550169 15138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:18:01.565006 15138 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:18:01.566247 15138 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:18:01.583084 15138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcb9cd11b8f14c779e8898ae366b225b: Generated new TSK 0
I20260620 20:18:01.584123 15138 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:18:01.614866 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:01.612757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.24.20.189:39433)
0620 20:18:01.612874 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.612878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.613749 (+ 871us) server_negotiation.cc:407] Connection header received
0620 20:18:01.613956 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.613960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.614013 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.614072 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.614718 (+ 646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36542: BlockingRecv error: recv got EOF from 127.0.0.1:36542 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 20:18:01.616221 15088 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:01.613139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.24.20.188:42239)
0620 20:18:01.613272 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.613276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.615354 (+ 2078us) server_negotiation.cc:407] Connection header received
0620 20:18:01.615534 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.615538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.615586 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.615637 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.616109 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34406: BlockingRecv error: recv got EOF from 127.0.0.1:34406 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 20:18:01.617568 15087 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:01.612341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.24.20.190:36127)
0620 20:18:01.612469 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.612474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.616703 (+ 4229us) server_negotiation.cc:407] Connection header received
0620 20:18:01.616865 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.616868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.616914 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.616965 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.617436 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38656: BlockingRecv error: recv got EOF from 127.0.0.1:38656 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 20:18:01.621508 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:01.620225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.24.20.189:39433)
0620 20:18:01.620330 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.620334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.620514 (+ 180us) server_negotiation.cc:407] Connection header received
0620 20:18:01.620782 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.620786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.620868 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.620919 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.621400 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36548: BlockingRecv error: recv got EOF from 127.0.0.1:36548 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 20:18:01.630285 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:01.629360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.24.20.189:39433)
0620 20:18:01.629475 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.629479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.629491 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:18:01.629543 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.629546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.629593 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.629680 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.630178 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36550: BlockingRecv error: recv got EOF from 127.0.0.1:36550 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 20:18:01.635766 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:01.634250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.24.20.189:39433)
0620 20:18:01.634360 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.634366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.634874 (+ 508us) server_negotiation.cc:407] Connection header received
0620 20:18:01.635067 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.635070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.635119 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.635164 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.635665 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36558: BlockingRecv error: recv got EOF from 127.0.0.1:36558 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 20:18:01.643625 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:01.641960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.24.20.189:39433)
0620 20:18:01.642101 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.642105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.642671 (+ 566us) server_negotiation.cc:407] Connection header received
0620 20:18:01.642871 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.642875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.642927 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.642984 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.643520 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36572: BlockingRecv error: recv got EOF from 127.0.0.1:36572 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 20:18:01.658391 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:01.654571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.24.20.189:39433)
0620 20:18:01.654691 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.654695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.655298 (+ 603us) server_negotiation.cc:407] Connection header received
0620 20:18:01.657553 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.657557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.657619 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.657676 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.658281 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36586: BlockingRecv error: recv got EOF from 127.0.0.1:36586 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 20:18:01.678674 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:01.676983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.24.20.189:39433)
0620 20:18:01.677100 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.677104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.677682 (+ 578us) server_negotiation.cc:407] Connection header received
0620 20:18:01.677916 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.677920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.677973 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.678025 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.678556 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36598: BlockingRecv error: recv got EOF from 127.0.0.1:36598 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 20:18:01.714632 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:01.713210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.24.20.189:39433)
0620 20:18:01.713344 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.713349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.713658 (+ 309us) server_negotiation.cc:407] Connection header received
0620 20:18:01.713843 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.713847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.713910 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.714021 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.714500 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36606: BlockingRecv error: recv got EOF from 127.0.0.1:36606 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 20:18:01.782217 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:01.780973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.24.20.189:39433)
0620 20:18:01.781136 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.781141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.781292 (+ 151us) server_negotiation.cc:407] Connection header received
0620 20:18:01.781474 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.781478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.781536 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.781633 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.782089 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36618: BlockingRecv error: recv got EOF from 127.0.0.1:36618 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 20:18:01.913821 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:01.912517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.24.20.189:39433)
0620 20:18:01.912713 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:01.912718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:01.912839 (+ 121us) server_negotiation.cc:407] Connection header received
0620 20:18:01.913034 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:01.913038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:01.913093 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:01.913190 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:01.913712 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36624: BlockingRecv error: recv got EOF from 127.0.0.1:36624 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260620 20:18:02.173341 15013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:18:02.172236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.24.20.189:39433)
0620 20:18:02.172383 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:02.172388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:02.172404 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:18:02.172519 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:02.172524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:02.172576 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:02.172733 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:18:02.173203 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36630: BlockingRecv error: recv got EOF from 127.0.0.1:36630 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:18:02.478247 15101 catalog_manager.cc:1259] Loaded cluster ID: e5d41a4cf9c2452594d8a04a2bcf7cef
I20260620 20:18:02.478338 15101 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f: loading cluster ID for follower catalog manager: success
I20260620 20:18:02.479974 15101 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f: acquiring CA information for follower catalog manager: success
I20260620 20:18:02.481138 15101 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ae4729ab354429d8def72f4c76eab0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:18:02.500896 15128 catalog_manager.cc:1259] Loaded cluster ID: e5d41a4cf9c2452594d8a04a2bcf7cef
I20260620 20:18:02.501010 15128 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480: loading cluster ID for follower catalog manager: success
I20260620 20:18:02.502630 15128 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480: acquiring CA information for follower catalog manager: success
I20260620 20:18:02.503786 15128 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ed0ad437c2f24085b8a7b25815786480: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:18:02.691737 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:02.687496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.24.20.189:39433)
0620 20:18:02.687755 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:02.687758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:02.687841 (+ 83us) server_negotiation.cc:407] Connection header received
0620 20:18:02.688125 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:02.688128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:02.688251 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:02.688350 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:02.688697 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:02.689218 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:02.690173 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:02.690390 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:02.690463 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:02.690798 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:02.691283 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:02.691372 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:02.691494 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 20:18:02.691541 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260620 20:18:02.698477 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:02.694759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.24.20.188:42239)
0620 20:18:02.695019 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:02.695026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:02.695042 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:18:02.695137 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:02.695139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:02.695247 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:02.695374 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:02.695592 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:02.696102 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:02.697001 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:02.697238 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:02.697354 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:02.697657 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:02.698061 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:02.698200 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:02.698260 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 20:18:02.698316 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260620 20:18:02.705567 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:02.701394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.24.20.189:39433)
0620 20:18:02.701581 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:02.701585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:02.701833 (+ 248us) server_negotiation.cc:407] Connection header received
0620 20:18:02.702002 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:02.702008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:02.702064 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:02.702103 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:02.702514 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:02.703305 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:02.704349 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:02.704709 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:02.704820 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:02.705049 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:02.705137 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:02.705207 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:02.705371 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 20:18:02.705433 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 20:18:02.705971 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:02.701852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.24.20.188:42239)
0620 20:18:02.701990 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:02.701994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:02.702147 (+ 153us) server_negotiation.cc:407] Connection header received
0620 20:18:02.702254 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:02.702257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:02.702302 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:02.702390 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:02.702652 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:02.703377 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:02.704548 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:02.704933 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:02.705057 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:02.705429 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:02.705517 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:02.705573 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:02.705765 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 20:18:02.705817 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:18:02.706202 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:18:02.701346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.24.20.190:36127)
0620 20:18:02.701611 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 20:18:02.701616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:18:02.701903 (+ 287us) server_negotiation.cc:407] Connection header received
0620 20:18:02.702299 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:18:02.702303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:18:02.702504 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:18:02.702642 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:18:02.702912 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:02.703586 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:02.704715 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:18:02.705067 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:18:02.705193 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:18:02.705494 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:18:02.705831 (+ 337us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:18:02.705925 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 20:18:02.706009 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 20:18:02.706066 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
I20260620 20:18:02.709565 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 14863
I20260620 20:18:02.716306 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 14938
I20260620 20:18:02.722011 24658 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVWkq7u/build/debug/bin/kudu with pid 15012
2026-06-20T20:18:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3060 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6148 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277087 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST